[jira] [Created] (MAPREDUCE-5533) Speculative execution does not function for reduce
Tassapol Athiapinya created MAPREDUCE-5533: -- Summary: Speculative execution does not function for reduce Key: MAPREDUCE-5533 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5533 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Tassapol Athiapinya Fix For: 2.1.1-beta We have sort job where reduce attempt does not send heartbeat in timely manner to application master. The AM should kick off another attempt to let job succeeds. What we find is the job fails and there is no speculation happening. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777057#comment-13777057 ] Hudson commented on MAPREDUCE-5505: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4464 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4464/]) MAPREDUCE-5505. Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526071) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen >Priority: Critical > Fix For: 2.1.2-beta > > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch, MAPREDUCE-5505.7.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777050#comment-13777050
]
Zhijie Shen commented on MAPREDUCE-5530:
The problem is valid, and the patch can ensure binary/source compatibility.
However, I've still some other concern. The incompatibility only happens when
the subclasses refer to protected boolean isSplitable(FileSystem fs, Path file)
or override it:
1. If the subclasses refer to isSplitable(FileSystem fs, Path file), the patch
provides the hook the subclasses.
2. If the subclasses override isSplitable(FileSystem fs, Path file),
NoSuchMethodError will not be thrown as well. However, since the method is
never called by getSplits() (assume that getSplits() is not overrode to call
this method instead), the override actually doesn't work, which may logically
break the user application.
In the second case, no immediate idea about how to deal with it.
> Binary and source incompatibility in mapred.lib.CombineFileInputFormat
> between branch-1 and branch-2
>
>
> Key: MAPREDUCE-5530
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5530.patch
>
>
> {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
> {code:java}
> protected boolean isSplitable(FileSystem fs, Path file)
> {code}
> In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
> {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
> similar method:
> {code:java}
> protected boolean isSplitable(JobContext context, Path file)
> {code}
> This means that any code that subclasses
> {{mapred.lib.CombineFileInputFormat}} and does not provide its own
> implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
> will not be binary or source compatible if it tries to call {{isSplitable}}
> with a {{FileSystem}} argument anywhere (that is, if compiled against
> branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
> also, it won't even compile against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5481) TestUberAM timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5481: --- Target Version/s: 2.3.0 > TestUberAM timeout > -- > > Key: MAPREDUCE-5481 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 3.0.0 >Reporter: Jason Lowe >Assignee: Xuan Gong >Priority: Blocker > Attachments: syslog > > > TestUberAM has been timing out on trunk for some time now and surefire then > fails the build. I'm not able to reproduce it locally, but the Jenkins > builds have been seeing it fairly consistently. See > https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-5505: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1. Committed to trunk, branch-2 and branch-2.1-beta > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen >Priority: Critical > Fix For: 2.1.2-beta > > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch, MAPREDUCE-5505.7.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-5505: -- Fix Version/s: 2.1.2-beta > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen > Fix For: 2.1.2-beta > > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch, MAPREDUCE-5505.7.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-5505: -- Priority: Critical (was: Major) > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen >Priority: Critical > Fix For: 2.1.2-beta > > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch, MAPREDUCE-5505.7.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777010#comment-13777010
]
Zhijie Shen commented on MAPREDUCE-5531:
+1 for the latest patch
> Binary and source incompatibility in mapreduce.TaskID and
> mapreduce.TaskAttemptID between branch-1 and branch-2
> ---
>
> Key: MAPREDUCE-5531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5531.patch, MAPREDUCE-5531.patch
>
>
> {{mapreduce.TaskID}} in branch-1 has these two constructors:
> {code:java}
> public TaskID(JobID jobId, boolean isMap, int id)
> public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
> {code}
> In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
> constructors.
> Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
> {code:java}
> public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
> taskId, int id)
> {code}
> In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
> It looks like these constructors were probably removed because the {{boolean
> isMap}} was replaced by an enum, {{TaskType}}.
> This means that any code that tries to use any of those constructors will not
> be binary or source compatible (in fact, the missing {{TaskAttemptID}}
> constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5532) Container localizer logs are not created in secured cluster
Omkar Vinit Joshi created MAPREDUCE-5532: Summary: Container localizer logs are not created in secured cluster Key: MAPREDUCE-5532 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5532 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Omkar Vinit Joshi Assignee: Omkar Vinit Joshi When we are running ContainerLocalizer in secured cluster we potentially are not creating any log file to track log messages. This will be helpful in potentially identifying ContainerLocalization issues in secured cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776994#comment-13776994
]
Zhijie Shen commented on MAPREDUCE-5529:
+1 for the latest patch
bq. Zhijie Shen, can you also look at MAPREDUCE-5530?
Will do that
> Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
> and branch-2
> --
>
> Key: MAPREDUCE-5529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5529.patch, MAPREDUCE-5529.patch
>
>
> {{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
> {code:java}
> public static String getPartitionFile(JobConf job)
> public static void setPartitionFile(JobConf job, Path p)
> {code}
> In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
> {{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
> methods:
> {code:java}
> public static String getPartitionFile(Configuration conf)
> public static void setPartitionFile(Configuration conf, Path p)
> {code}
> This means that any code that does either of the following:
> {code:java}
> TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
> String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
> {code}
> will not be binary compatible (that is, if compiled against branch-1, it will
> throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776995#comment-13776995
]
Hadoop QA commented on MAPREDUCE-5529:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604924/MAPREDUCE-5529.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//console
This message is automatically generated.
> Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
> and branch-2
> --
>
> Key: MAPREDUCE-5529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5529.patch, MAPREDUCE-5529.patch
>
>
> {{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
> {code:java}
> public static String getPartitionFile(JobConf job)
> public static void setPartitionFile(JobConf job, Path p)
> {code}
> In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
> {{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
> methods:
> {code:java}
> public static String getPartitionFile(Configuration conf)
> public static void setPartitionFile(Configuration conf, Path p)
> {code}
> This means that any code that does either of the following:
> {code:java}
> TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
> String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
> {code}
> will not be binary compatible (that is, if compiled against branch-1, it will
> throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776985#comment-13776985 ] Zhijie Shen commented on MAPREDUCE-5505: Again, the test failure is unrelated. > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch, MAPREDUCE-5505.7.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[
https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776978#comment-13776978
]
Hadoop QA commented on MAPREDUCE-5505:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604920/MAPREDUCE-5505.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4038//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4038//console
This message is automatically generated.
> Clients should be notified job finished only after job successfully
> unregistered
> -
>
> Key: MAPREDUCE-5505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Zhijie Shen
> Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch,
> MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch,
> MAPREDUCE-5505.6.patch, MAPREDUCE-5505.7.patch
>
>
> This is to make sure user is notified job finished after job is really done.
> This does increase client latency but can reduce some races during unregister
> like YARN-540
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776972#comment-13776972
]
Hadoop QA commented on MAPREDUCE-5531:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604912/MAPREDUCE-5531.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4039//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4039//console
This message is automatically generated.
> Binary and source incompatibility in mapreduce.TaskID and
> mapreduce.TaskAttemptID between branch-1 and branch-2
> ---
>
> Key: MAPREDUCE-5531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5531.patch, MAPREDUCE-5531.patch
>
>
> {{mapreduce.TaskID}} in branch-1 has these two constructors:
> {code:java}
> public TaskID(JobID jobId, boolean isMap, int id)
> public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
> {code}
> In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
> constructors.
> Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
> {code:java}
> public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
> taskId, int id)
> {code}
> In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
> It looks like these constructors were probably removed because the {{boolean
> isMap}} was replaced by an enum, {{TaskType}}.
> This means that any code that tries to use any of those constructors will not
> be binary or source compatible (in fact, the missing {{TaskAttemptID}}
> constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5529:
-
Attachment: MAPREDUCE-5529.patch
The new patch marks the methods as deprecated and adds javadoc.
[~zjshen], can you also look at MAPREDUCE-5530? It's another compatibility
issue (I created 3: MAPREDUCE-5529, MAPREDUCE-5530, and MAPREDUCE-5531) thanks
> Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
> and branch-2
> --
>
> Key: MAPREDUCE-5529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5529.patch, MAPREDUCE-5529.patch
>
>
> {{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
> {code:java}
> public static String getPartitionFile(JobConf job)
> public static void setPartitionFile(JobConf job, Path p)
> {code}
> In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
> {{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
> methods:
> {code:java}
> public static String getPartitionFile(Configuration conf)
> public static void setPartitionFile(Configuration conf, Path p)
> {code}
> This means that any code that does either of the following:
> {code:java}
> TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
> String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
> {code}
> will not be binary compatible (that is, if compiled against branch-1, it will
> throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5514) TestRMContainerAllocator fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5514: --- Target Version/s: 2.1.2-beta > TestRMContainerAllocator fails on trunk > --- > > Key: MAPREDUCE-5514 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5514 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > Attachments: MAPREDUCE-5514.1.patch, > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator-output.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5505: --- Status: Patch Available (was: Open) > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch, MAPREDUCE-5505.7.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[
https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776938#comment-13776938
]
Hadoop QA commented on MAPREDUCE-5505:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604909/MAPREDUCE-5505.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4037//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4037//console
This message is automatically generated.
> Clients should be notified job finished only after job successfully
> unregistered
> -
>
> Key: MAPREDUCE-5505
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Zhijie Shen
> Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch,
> MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch,
> MAPREDUCE-5505.6.patch, MAPREDUCE-5505.7.patch
>
>
> This is to make sure user is notified job finished after job is really done.
> This does increase client latency but can reduce some races during unregister
> like YARN-540
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5505: --- Attachment: MAPREDUCE-5505.7.patch The new patch gives the default value to lastNonFinalState, and recovers some tests that are missed in the last patch > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch, MAPREDUCE-5505.7.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5505: --- Status: Open (was: Patch Available) > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5505: --- Target Version/s: 2.1.2-beta > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5531:
-
Attachment: MAPREDUCE-5531.patch
The new patch marks the 3 constructors as deprecated.
> Binary and source incompatibility in mapreduce.TaskID and
> mapreduce.TaskAttemptID between branch-1 and branch-2
> ---
>
> Key: MAPREDUCE-5531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5531.patch, MAPREDUCE-5531.patch
>
>
> {{mapreduce.TaskID}} in branch-1 has these two constructors:
> {code:java}
> public TaskID(JobID jobId, boolean isMap, int id)
> public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
> {code}
> In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
> constructors.
> Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
> {code:java}
> public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
> taskId, int id)
> {code}
> In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
> It looks like these constructors were probably removed because the {{boolean
> isMap}} was replaced by an enum, {{TaskType}}.
> This means that any code that tries to use any of those constructors will not
> be binary or source compatible (in fact, the missing {{TaskAttemptID}}
> constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5530:
-
Issue Type: Sub-task (was: Bug)
Parent: MAPREDUCE-5108
> Binary and source incompatibility in mapred.lib.CombineFileInputFormat
> between branch-1 and branch-2
>
>
> Key: MAPREDUCE-5530
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5530.patch
>
>
> {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
> {code:java}
> protected boolean isSplitable(FileSystem fs, Path file)
> {code}
> In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
> {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
> similar method:
> {code:java}
> protected boolean isSplitable(JobContext context, Path file)
> {code}
> This means that any code that subclasses
> {{mapred.lib.CombineFileInputFormat}} and does not provide its own
> implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
> will not be binary or source compatible if it tries to call {{isSplitable}}
> with a {{FileSystem}} argument anywhere (that is, if compiled against
> branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
> also, it won't even compile against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5530:
-
Fix Version/s: 2.1.2-beta
> Binary and source incompatibility in mapred.lib.CombineFileInputFormat
> between branch-1 and branch-2
>
>
> Key: MAPREDUCE-5530
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Fix For: 2.1.2-beta
>
> Attachments: MAPREDUCE-5530.patch
>
>
> {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
> {code:java}
> protected boolean isSplitable(FileSystem fs, Path file)
> {code}
> In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
> {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
> similar method:
> {code:java}
> protected boolean isSplitable(JobContext context, Path file)
> {code}
> This means that any code that subclasses
> {{mapred.lib.CombineFileInputFormat}} and does not provide its own
> implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
> will not be binary or source compatible if it tries to call {{isSplitable}}
> with a {{FileSystem}} argument anywhere (that is, if compiled against
> branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
> also, it won't even compile against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5530:
-
Target Version/s: 2.1.2-beta
Fix Version/s: (was: 2.1.2-beta)
> Binary and source incompatibility in mapred.lib.CombineFileInputFormat
> between branch-1 and branch-2
>
>
> Key: MAPREDUCE-5530
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5530.patch
>
>
> {{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
> {code:java}
> protected boolean isSplitable(FileSystem fs, Path file)
> {code}
> In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
> {{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
> similar method:
> {code:java}
> protected boolean isSplitable(JobContext context, Path file)
> {code}
> This means that any code that subclasses
> {{mapred.lib.CombineFileInputFormat}} and does not provide its own
> implementation of {{protected boolean isSplitable(FileSystem fs, Path file)}}
> will not be binary or source compatible if it tries to call {{isSplitable}}
> with a {{FileSystem}} argument anywhere (that is, if compiled against
> branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2;
> also, it won't even compile against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated MAPREDUCE-5531:
---
Target Version/s: 2.1.2-beta
> Binary and source incompatibility in mapreduce.TaskID and
> mapreduce.TaskAttemptID between branch-1 and branch-2
> ---
>
> Key: MAPREDUCE-5531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5531.patch
>
>
> {{mapreduce.TaskID}} in branch-1 has these two constructors:
> {code:java}
> public TaskID(JobID jobId, boolean isMap, int id)
> public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
> {code}
> In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
> constructors.
> Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
> {code:java}
> public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
> taskId, int id)
> {code}
> In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
> It looks like these constructors were probably removed because the {{boolean
> isMap}} was replaced by an enum, {{TaskType}}.
> This means that any code that tries to use any of those constructors will not
> be binary or source compatible (in fact, the missing {{TaskAttemptID}}
> constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5505: --- Attachment: MAPREDUCE-5505.6.patch The new patch addressed the last Bikas' comments. [~bikassaha], would you please have a look again? > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch, > MAPREDUCE-5505.6.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776894#comment-13776894
]
Zhijie Shen commented on MAPREDUCE-5529:
[~rkanter], like MAPREDUCE-5531, would you please deprecate the added methods,
and add the javadoc to pointer the users to the new methods?
> Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
> and branch-2
> --
>
> Key: MAPREDUCE-5529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5529.patch
>
>
> {{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
> {code:java}
> public static String getPartitionFile(JobConf job)
> public static void setPartitionFile(JobConf job, Path p)
> {code}
> In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
> {{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
> methods:
> {code:java}
> public static String getPartitionFile(Configuration conf)
> public static void setPartitionFile(Configuration conf, Path p)
> {code}
> This means that any code that does either of the following:
> {code:java}
> TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
> String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
> {code}
> will not be binary compatible (that is, if compiled against branch-1, it will
> throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5488) Job recovery fails after killing all the running containers for the app
[
https://issues.apache.org/jira/browse/MAPREDUCE-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-5488:
---
Fix Version/s: 2.1.2-beta
> Job recovery fails after killing all the running containers for the app
> ---
>
> Key: MAPREDUCE-5488
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5488
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Arpit Gupta
>Assignee: Jian He
> Fix For: 2.1.2-beta
>
> Attachments: MAPREDUCE-5488.1.patch, MAPREDUCE-5488.2.patch,
> MAPREDUCE-5488.3.patch, MAPREDUCE-5488.patch, MAPREDUCE-5488.patch,
> MAPREDUCE-5488.patch
>
>
> Here is the client stack trace
> {code}
> RUNNING: /usr/lib/hadoop/bin/hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.1.0.2.0.5.0-66.jar
> wordcount "-Dmapreduce.reduce.input.limit=-1"
> /user/user/test_yarn_ha/medium_wordcount_input
> /user/hrt_qa/test_yarn_ha/test_mapred_ha_single_job_applicationmaster-1-time
> 13/08/30 08:45:39 INFO client.RMProxy: Connecting to ResourceManager at
> hostname/68.142.247.148:8032
> 13/08/30 08:45:40 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 19
> for user on ha-hdfs:ha-2-secure
> 13/08/30 08:45:40 INFO security.TokenCache: Got dt for hdfs://ha-2-secure;
> Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ha-2-secure, Ident:
> (HDFS_DELEGATION_TOKEN token 19 for user)
> 13/08/30 08:45:40 INFO input.FileInputFormat: Total input paths to process :
> 20
> 13/08/30 08:45:40 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 13/08/30 08:45:40 INFO lzo.LzoCodec: Successfully loaded & initialized
> native-lzo library [hadoop-lzo rev cf4e7cbf8ed0f0622504d008101c2729dc0c9ff3]
> 13/08/30 08:45:40 INFO mapreduce.JobSubmitter: number of splits:180
> 13/08/30 08:45:40 WARN conf.Configuration: user.name is deprecated. Instead,
> use mapreduce.job.user.name
> 13/08/30 08:45:40 WARN conf.Configuration: mapred.jar is deprecated. Instead,
> use mapreduce.job.jar
> 13/08/30 08:45:40 WARN conf.Configuration: mapred.output.value.class is
> deprecated. Instead, use mapreduce.job.output.value.class
> 13/08/30 08:45:40 WARN conf.Configuration: mapreduce.combine.class is
> deprecated. Instead, use mapreduce.job.combine.class
> 13/08/30 08:45:40 WARN conf.Configuration: mapreduce.map.class is deprecated.
> Instead, use mapreduce.job.map.class
> 13/08/30 08:45:40 WARN conf.Configuration: mapred.job.name is deprecated.
> Instead, use mapreduce.job.name
> 13/08/30 08:45:40 WARN conf.Configuration: mapreduce.reduce.class is
> deprecated. Instead, use mapreduce.job.reduce.class
> 13/08/30 08:45:40 WARN conf.Configuration: mapred.input.dir is deprecated.
> Instead, use mapreduce.input.fileinputformat.inputdir
> 13/08/30 08:45:40 WARN conf.Configuration: mapred.output.dir is deprecated.
> Instead, use mapreduce.output.fileoutputformat.outputdir
> 13/08/30 08:45:40 WARN conf.Configuration: mapred.map.tasks is deprecated.
> Instead, use mapreduce.job.maps
> 13/08/30 08:45:40 WARN conf.Configuration: mapred.output.key.class is
> deprecated. Instead, use mapreduce.job.output.key.class
> 13/08/30 08:45:40 WARN conf.Configuration: mapred.working.dir is deprecated.
> Instead, use mapreduce.job.working.dir
> 13/08/30 08:45:41 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1377851032086_0003
> 13/08/30 08:45:41 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN,
> Service: ha-hdfs:ha-2-secure, Ident: (HDFS_DELEGATION_TOKEN token 19 for user)
> 13/08/30 08:45:42 INFO impl.YarnClientImpl: Submitted application
> application_1377851032086_0003 to ResourceManager at
> hostname/68.142.247.148:8032
> 13/08/30 08:45:42 INFO mapreduce.Job: The url to track the job:
> http://hostname:8088/proxy/application_1377851032086_0003/
> 13/08/30 08:45:42 INFO mapreduce.Job: Running job: job_1377851032086_0003
> 13/08/30 08:45:48 INFO mapreduce.Job: Job job_1377851032086_0003 running in
> uber mode : false
> 13/08/30 08:45:48 INFO mapreduce.Job: map 0% reduce 0%
> stop applicationmaster
> beaver.component.hadoop|INFO|Kill container
> container_1377851032086_0003_01_01 on host hostname
> RUNNING: ssh -o StrictHostKeyChecking=no hostname "sudo su - -c \"ps aux |
> grep container_1377851032086_0003_01_01 | awk '{print \\\$2}' | xargs
> kill -9\" root"
> Warning: Permanently added 'hostname,68.142.247.155' (RSA) to the list of
> known hosts.
> kill 8978: No such process
> waiting for down time 10 seconds for service applicationmaster
> 13/08/30 08:45:55 INFO ipc.Client: Retrying connect to server:
> hostname/68.142.247.155:52713. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1 SECONDS)
> 13/08/3
[jira] [Commented] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776889#comment-13776889
]
Zhijie Shen commented on MAPREDUCE-5531:
[~rkanter], thanks for finding the issue. The patch looks almost good, but
would you please mark the added constructor deprecated?
> Binary and source incompatibility in mapreduce.TaskID and
> mapreduce.TaskAttemptID between branch-1 and branch-2
> ---
>
> Key: MAPREDUCE-5531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5531.patch
>
>
> {{mapreduce.TaskID}} in branch-1 has these two constructors:
> {code:java}
> public TaskID(JobID jobId, boolean isMap, int id)
> public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
> {code}
> In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
> constructors.
> Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
> {code:java}
> public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
> taskId, int id)
> {code}
> In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
> It looks like these constructors were probably removed because the {{boolean
> isMap}} was replaced by an enum, {{TaskType}}.
> This means that any code that tries to use any of those constructors will not
> be binary or source compatible (in fact, the missing {{TaskAttemptID}}
> constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated MAPREDUCE-5531:
---
Issue Type: Sub-task (was: Bug)
Parent: MAPREDUCE-5108
> Binary and source incompatibility in mapreduce.TaskID and
> mapreduce.TaskAttemptID between branch-1 and branch-2
> ---
>
> Key: MAPREDUCE-5531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5531.patch
>
>
> {{mapreduce.TaskID}} in branch-1 has these two constructors:
> {code:java}
> public TaskID(JobID jobId, boolean isMap, int id)
> public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
> {code}
> In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
> constructors.
> Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
> {code:java}
> public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
> taskId, int id)
> {code}
> In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
> It looks like these constructors were probably removed because the {{boolean
> isMap}} was replaced by an enum, {{TaskType}}.
> This means that any code that tries to use any of those constructors will not
> be binary or source compatible (in fact, the missing {{TaskAttemptID}}
> constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5515) Application Manager UI does not appear with Https enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-5515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5515: --- Fix Version/s: 2.1.2-beta > Application Manager UI does not appear with Https enabled > - > > Key: MAPREDUCE-5515 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5515 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Fix For: 2.1.2-beta > > Attachments: MAPREDUCE-5515.txt > > > related issue YARN-1203. We need to disable https for MR-AM by default as > they will need access to keystore which can not be granted in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5523) Need to add https port related property in Job history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5523: --- Fix Version/s: 2.1.2-beta > Need to add https port related property in Job history server > - > > Key: MAPREDUCE-5523 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5523 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Fix For: 2.1.2-beta > > Attachments: MAPREDUCE-5523.20131020.1.patch, > MAPREDUCE-5523.20131020.2.patch > > > related ticket YARN-1204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5503) TestMRJobClient.testJobClient is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5503:
-
Target Version/s: 3.0.0, 2.1.2-beta (was: 3.0.0, 2.1.1-beta)
> TestMRJobClient.testJobClient is failing
>
>
> Key: MAPREDUCE-5503
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5503
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Jason Lowe
>Assignee: Jian He
>Priority: Blocker
> Attachments: MAPREDUCE-5503.patch, MAPREDUCE-5503.patch
>
>
> TestMRJobClient.testJobClient is failing on trunk and causing precommit
> builds to complain:
> {noformat}
> testJobClient(org.apache.hadoop.mapreduce.TestMRJobClient) Time elapsed:
> 26.361 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<1> but was:<0>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at junit.framework.Assert.assertEquals(Assert.java:205)
> at
> org.apache.hadoop.mapreduce.TestMRJobClient.testJobList(TestMRJobClient.java:474)
> at
> org.apache.hadoop.mapreduce.TestMRJobClient.testJobClient(TestMRJobClient.java:112)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-5529:
-
Target Version/s: 2.1.2-beta
> Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
> and branch-2
> --
>
> Key: MAPREDUCE-5529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5529.patch
>
>
> {{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
> {code:java}
> public static String getPartitionFile(JobConf job)
> public static void setPartitionFile(JobConf job, Path p)
> {code}
> In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
> {{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
> methods:
> {code:java}
> public static String getPartitionFile(Configuration conf)
> public static void setPartitionFile(Configuration conf, Path p)
> {code}
> This means that any code that does either of the following:
> {code:java}
> TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
> String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
> {code}
> will not be binary compatible (that is, if compiled against branch-1, it will
> throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5505) Clients should be notified job finished only after job successfully unregistered
[ https://issues.apache.org/jira/browse/MAPREDUCE-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776801#comment-13776801 ] Bikas Saha commented on MAPREDUCE-5505: --- Instead of calling rememberLastState in every transition (which is also not future proof) can we call it in JobImpl.handle(). This is where the transition method gets called. Similarly, createRunningStubbedJob() can implicitly create a mockContext that returns true. Only in the relevant tests, we can pass in/override the mockContext to return false. This will reduce the code change. Rest looks good to me. > Clients should be notified job finished only after job successfully > unregistered > - > > Key: MAPREDUCE-5505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5505 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jian He >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5505.1.patch, MAPREDUCE-5505.1.patch, > MAPREDUCE-5505.3.patch, MAPREDUCE-5505.4.patch, MAPREDUCE-5505.5.patch > > > This is to make sure user is notified job finished after job is really done. > This does increase client latency but can reduce some races during unregister > like YARN-540 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[
https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776793#comment-13776793
]
Avner BenHanoch commented on MAPREDUCE-5329:
let me summarize what I understand for making sure we are on the same page:
1. you still want "mapreduce.job.shuffle.provider.plugin.classes" (that I don't
like)
2. I'll remove this propery from mapred-default.xml
3. I'll add a comment next to serviceData.put saying that this needs to be able
to work with the host:port ...
now, for the rest:
{quote}
Shouldn't an attempt be made to initialize plugins which are specified by the
job ? Walking over configured NM plugins ends up potentially skipping plugins
which the job may have configured, but are not running in the NodeManager.
{quote}
In this case, I don't know how to calculate serviceName from serviceClass. I
think we'll need to change the config to contain serviceNames instead of
serviceClasses. Is this what you mean?
> APPLICATION_INIT is never sent to AuxServices other than the builtin
> ShuffleHandler
> ---
>
> Key: MAPREDUCE-5329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.1.0-beta, 2.0.6-alpha
>Reporter: Avner BenHanoch
> Fix For: trunk
>
> Attachments: MAPREDUCE-5329.patch
>
>
> APPLICATION_INIT is never sent to AuxServices other than the built-in
> ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be
> able to function, because APPLICATION_INIT enables the AuxiliaryService to
> map jobId->userId. This is needed for properly finding the MOFs of a job per
> reducers' requests.
> NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to
> hard-coded expression in hadoop code. The current TaskAttemptImpl.java code
> explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID,
> ...) and ignores any additional AuxiliaryService. As a result, only the
> built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party
> AuxillaryService will never get APPLICATION_INIT events.
> I think a solution can be in one of two ways:
> 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register
> each of them, by calling serviceData.put (…) in loop.
> 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668
> "APPLICATION_STOP is never sent to AuxServices". This means that in case the
> 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux
> Services regardless of the value in event.getServiceID().
> I prefer the 2nd solution. I am welcoming any ideas. I can provide the
> needed patch for any option that people like.
> See [Pluggable Shuffle in Hadoop
> documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3859) CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776788#comment-13776788 ] Mike Roark commented on MAPREDUCE-3859: --- I recently tested this issue in CDH4.2 to determine if we needed to patch it. I couldn't reproduce it, so it must have gotten fixed by that version. > CapacityScheduler incorrectly utilizes extra-resources of queue for > high-memory jobs > > > Key: MAPREDUCE-3859 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3859 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: capacity-sched >Affects Versions: 1.0.0 >Reporter: Sergey Tryuber >Assignee: Sergey Tryuber > Fix For: 1.2.1 > > Attachments: MAPREDUCE-3859_MR1_fix_and_test.patch.txt, > test-to-fail.patch.txt > > > Imagine, we have a queue A with capacity 10 slots and 20 as extra-capacity, > jobs which use 3 map slots will never consume more than 9 slots, regardless > how many free slots on a cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776736#comment-13776736 ] Siddharth Seth commented on MAPREDUCE-5329: --- bq. It is true that shuffle-consumer is at job level. However, here we are dealing with shuffle-provider which is an aux-service that serves multiple jobs. Hence, it is not at job level. An AM is per job - and the config being used is really a per AM/job configuration. The config is just saying that this particular job may use an alternate shuffle mechanism - but isn't related to what services will be run on the cluster. The NM would have this configured at a cluster level (equivalent to the TT config). bq. last comment, we already have shuffle.consumer.plugin in mapred-site.xml, wouldn't it will be appropriate to have the shuffle.provider.plugin next to it. This is not critical for me, just let me know what you prefer. I prefer leaving it out. Also it'd be good to have a comment which says that this needs to be able to work with the host:port information provided by the AM (i.e. plugins which require custom location / other configuration are not supported) bq. I'll surround it with ... Shouldn't an attempt be made to initialize plugins which are specified by the job ? Walking over configured NM plugins ends up potentially skipping plugins which the job may have configured, but are not running in the NodeManager. > APPLICATION_INIT is never sent to AuxServices other than the builtin > ShuffleHandler > --- > > Key: MAPREDUCE-5329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.1.0-beta, 2.0.6-alpha >Reporter: Avner BenHanoch > Fix For: trunk > > Attachments: MAPREDUCE-5329.patch > > > APPLICATION_INIT is never sent to AuxServices other than the built-in > ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be > able to function, because APPLICATION_INIT enables the AuxiliaryService to > map jobId->userId. This is needed for properly finding the MOFs of a job per > reducers' requests. > NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to > hard-coded expression in hadoop code. The current TaskAttemptImpl.java code > explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID, > ...) and ignores any additional AuxiliaryService. As a result, only the > built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party > AuxillaryService will never get APPLICATION_INIT events. > I think a solution can be in one of two ways: > 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register > each of them, by calling serviceData.put (…) in loop. > 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668 > "APPLICATION_STOP is never sent to AuxServices". This means that in case the > 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux > Services regardless of the value in event.getServiceID(). > I prefer the 2nd solution. I am welcoming any ideas. I can provide the > needed patch for any option that people like. > See [Pluggable Shuffle in Hadoop > documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5522) Incorrectly expect the array of JobQueueInfo returned by o.a.h.mapred.QueueManager#getJobQueueInfos to have a specific order.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776706#comment-13776706 ] Robert Joseph Evans commented on MAPREDUCE-5522: OK let me know if your development needs it to go to a different line, and I'll try to merge it in. > Incorrectly expect the array of JobQueueInfo returned by > o.a.h.mapred.QueueManager#getJobQueueInfos to have a specific order. > - > > Key: MAPREDUCE-5522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 & IBM Java 1.6 >Reporter: Jinghui Wang >Assignee: Jinghui Wang >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: MAPREDUCE-5522.patch > > > There is a bug in test o.a.h.mapred.TestQueue. The implementation of > getJobQueueInfos in QueueManager uses the keySet of a HashMap to populate the > return value and since there is no guarantee in the ordering of the elements > in the keySet of a Hashmap, this test would fail if the order returned by > getJobQueueInfos is different than what the test is expecting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5513) ConcurrentModificationException in JobControl
[
https://issues.apache.org/jira/browse/MAPREDUCE-5513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776670#comment-13776670
]
Hadoop QA commented on MAPREDUCE-5513:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604863/MAPREDUCE-5513-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4036//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4036//console
This message is automatically generated.
> ConcurrentModificationException in JobControl
> -
>
> Key: MAPREDUCE-5513
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5513
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.1.0-beta, 0.23.9
>Reporter: Jason Lowe
>Assignee: Robert Parker
> Attachments: MAPREDUCE-5513-1.patch
>
>
> JobControl.toList is locking individual lists to iterate them, but those
> lists can be modified elsewhere without holding the list lock. The locking
> approaches are mismatched, with toList holding the lock on the actual list
> object while other methods hold the JobControl lock when modifying the lists.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5513) ConcurrentModificationException in JobControl
[ https://issues.apache.org/jira/browse/MAPREDUCE-5513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-5513: - Status: Patch Available (was: Open) > ConcurrentModificationException in JobControl > - > > Key: MAPREDUCE-5513 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5513 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.9, 2.1.0-beta >Reporter: Jason Lowe >Assignee: Robert Parker > Attachments: MAPREDUCE-5513-1.patch > > > JobControl.toList is locking individual lists to iterate them, but those > lists can be modified elsewhere without holding the list lock. The locking > approaches are mismatched, with toList holding the lock on the actual list > object while other methods hold the JobControl lock when modifying the lists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5513) ConcurrentModificationException in JobControl
[ https://issues.apache.org/jira/browse/MAPREDUCE-5513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-5513: - Attachment: MAPREDUCE-5513-1.patch > ConcurrentModificationException in JobControl > - > > Key: MAPREDUCE-5513 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5513 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.0-beta, 0.23.9 >Reporter: Jason Lowe >Assignee: Robert Parker > Attachments: MAPREDUCE-5513-1.patch > > > JobControl.toList is locking individual lists to iterate them, but those > lists can be modified elsewhere without holding the list lock. The locking > approaches are mismatched, with toList holding the lock on the actual list > object while other methods hold the JobControl lock when modifying the lists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[
https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776648#comment-13776648
]
Avner BenHanoch commented on MAPREDUCE-5329:
very last comment,
{quote}
Should the loading walk over the specified ShuffleProvider list, rather than
the ones available on the NodeManager
{quote}
I found it difficult to do it, because it is easy to get service-class-name
from service-name, but the opposite direction is harder. Do you mind if I'll
leave the loading as is, BUT I'll surround it with
{code}
if (! shuffleProviders.isEmpty())
{code}
> APPLICATION_INIT is never sent to AuxServices other than the builtin
> ShuffleHandler
> ---
>
> Key: MAPREDUCE-5329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.1.0-beta, 2.0.6-alpha
>Reporter: Avner BenHanoch
> Fix For: trunk
>
> Attachments: MAPREDUCE-5329.patch
>
>
> APPLICATION_INIT is never sent to AuxServices other than the built-in
> ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be
> able to function, because APPLICATION_INIT enables the AuxiliaryService to
> map jobId->userId. This is needed for properly finding the MOFs of a job per
> reducers' requests.
> NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to
> hard-coded expression in hadoop code. The current TaskAttemptImpl.java code
> explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID,
> ...) and ignores any additional AuxiliaryService. As a result, only the
> built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party
> AuxillaryService will never get APPLICATION_INIT events.
> I think a solution can be in one of two ways:
> 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register
> each of them, by calling serviceData.put (…) in loop.
> 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668
> "APPLICATION_STOP is never sent to AuxServices". This means that in case the
> 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux
> Services regardless of the value in event.getServiceID().
> I prefer the 2nd solution. I am welcoming any ideas. I can provide the
> needed patch for any option that people like.
> See [Pluggable Shuffle in Hadoop
> documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5517) enabling uber mode with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776588#comment-13776588 ] Vrushali C commented on MAPREDUCE-5517: --- Although there was a timeout, the test has passed (as seen in console output). Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.742 sec - in org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup Running org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator > enabling uber mode with 0 reducer still requires mapreduce.reduce.memory.mb > to be less than yarn.app.mapreduce.am.resource.mb > - > > Key: MAPREDUCE-5517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5517 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.5-alpha >Reporter: Siqi Li >Priority: Minor > Attachments: MAPREDUCE_5517_v3.patch.txt > > > Since there is no reducer, the memory allocated to reducer is irrelevant to > enable uber mode of a job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776561#comment-13776561
]
Hadoop QA commented on MAPREDUCE-5531:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604843/MAPREDUCE-5531.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4035//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4035//console
This message is automatically generated.
> Binary and source incompatibility in mapreduce.TaskID and
> mapreduce.TaskAttemptID between branch-1 and branch-2
> ---
>
> Key: MAPREDUCE-5531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5531.patch
>
>
> {{mapreduce.TaskID}} in branch-1 has these two constructors:
> {code:java}
> public TaskID(JobID jobId, boolean isMap, int id)
> public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
> {code}
> In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
> constructors.
> Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
> {code:java}
> public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
> taskId, int id)
> {code}
> In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
> It looks like these constructors were probably removed because the {{boolean
> isMap}} was replaced by an enum, {{TaskType}}.
> This means that any code that tries to use any of those constructors will not
> be binary or source compatible (in fact, the missing {{TaskAttemptID}}
> constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5531:
-
Status: Patch Available (was: Open)
> Binary and source incompatibility in mapreduce.TaskID and
> mapreduce.TaskAttemptID between branch-1 and branch-2
> ---
>
> Key: MAPREDUCE-5531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5531.patch
>
>
> {{mapreduce.TaskID}} in branch-1 has these two constructors:
> {code:java}
> public TaskID(JobID jobId, boolean isMap, int id)
> public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
> {code}
> In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
> constructors.
> Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
> {code:java}
> public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
> taskId, int id)
> {code}
> In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
> It looks like these constructors were probably removed because the {{boolean
> isMap}} was replaced by an enum, {{TaskType}}.
> This means that any code that tries to use any of those constructors will not
> be binary or source compatible (in fact, the missing {{TaskAttemptID}}
> constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-5531:
-
Attachment: MAPREDUCE-5531.patch
The patch adds the missing constructors, which simply call the other
constructors.
Unit tests aren't really possible for this.
> Binary and source incompatibility in mapreduce.TaskID and
> mapreduce.TaskAttemptID between branch-1 and branch-2
> ---
>
> Key: MAPREDUCE-5531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5531.patch
>
>
> {{mapreduce.TaskID}} in branch-1 has these two constructors:
> {code:java}
> public TaskID(JobID jobId, boolean isMap, int id)
> public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
> {code}
> In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
> constructors.
> Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
> {code:java}
> public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int
> taskId, int id)
> {code}
> In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
> It looks like these constructors were probably removed because the {{boolean
> isMap}} was replaced by an enum, {{TaskType}}.
> This means that any code that tries to use any of those constructors will not
> be binary or source compatible (in fact, the missing {{TaskAttemptID}}
> constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5531) Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2
Robert Kanter created MAPREDUCE-5531:
Summary: Binary and source incompatibility in mapreduce.TaskID and
mapreduce.TaskAttemptID between branch-1 and branch-2
Key: MAPREDUCE-5531
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5531
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
{{mapreduce.TaskID}} in branch-1 has these two constructors:
{code:java}
public TaskID(JobID jobId, boolean isMap, int id)
public TaskID(String jtIdentifier, int jobId, boolean isMap, int id)
{code}
In branch-2, {{mapreduce.TaskID}} no longer has either of the above two
constructors.
Also, {{mapreduce.TaskAttemptID}} in branch-1 has this constructor:
{code:java}
public TaskAttemptID(String jtIdentifier, int jobId, boolean isMap, int taskId,
int id)
{code}
In branch-2, {{mapreduce.TaskAttemptID}} no longer his this constructor.
It looks like these constructors were probably removed because the {{boolean
isMap}} was replaced by an enum, {{TaskType}}.
This means that any code that tries to use any of those constructors will not
be binary or source compatible (in fact, the missing {{TaskAttemptID}}
constructor calls one of the missing {{TaskID}} constructors).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5523) Need to add https port related property in Job history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776516#comment-13776516 ] Hudson commented on MAPREDUCE-5523: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4462 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4462/]) YARN-1204. Added separate configuration properties for https for RM and NM without which servers enabled with https will also start on http ports. Contributed by Omkar Vinit Joshi. MAPREDUCE-5523. Added separate configuration properties for https for JHS without which even when https is enabled, it starts on http port itself. Contributed by Omkar Vinit Joshi. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525947) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/WebAppUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebAppUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/amfilter/AmFilterInitializer.java > Need to add https port related property in Job history server > - > > Key: MAPREDUCE-5523 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5523 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: MAPREDUCE-5523.20131020.1.patch, > MAPREDUCE-5523.20131020.2.patch > > > related ticket YARN-1204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5523) Need to add https port related property in Job history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5523: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk, branch-2 and branch-2.1. Thanks Omkar! > Need to add https port related property in Job history server > - > > Key: MAPREDUCE-5523 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5523 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > Attachments: MAPREDUCE-5523.20131020.1.patch, > MAPREDUCE-5523.20131020.2.patch > > > related ticket YARN-1204 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776378#comment-13776378 ] Avner BenHanoch commented on MAPREDUCE-5329: last comment, we already have shuffle.consumer.plugin in mapred-site.xml, wouldn't it will be appropriate to have the shuffle.provider.plugin next to it. This is not critical for me, just let me know what you prefer. > APPLICATION_INIT is never sent to AuxServices other than the builtin > ShuffleHandler > --- > > Key: MAPREDUCE-5329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.1.0-beta, 2.0.6-alpha >Reporter: Avner BenHanoch > Fix For: trunk > > Attachments: MAPREDUCE-5329.patch > > > APPLICATION_INIT is never sent to AuxServices other than the built-in > ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be > able to function, because APPLICATION_INIT enables the AuxiliaryService to > map jobId->userId. This is needed for properly finding the MOFs of a job per > reducers' requests. > NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to > hard-coded expression in hadoop code. The current TaskAttemptImpl.java code > explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID, > ...) and ignores any additional AuxiliaryService. As a result, only the > built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party > AuxillaryService will never get APPLICATION_INIT events. > I think a solution can be in one of two ways: > 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register > each of them, by calling serviceData.put (…) in loop. > 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668 > "APPLICATION_STOP is never sent to AuxServices". This means that in case the > 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux > Services regardless of the value in event.getServiceID(). > I prefer the 2nd solution. I am welcoming any ideas. I can provide the > needed patch for any option that people like. > See [Pluggable Shuffle in Hadoop > documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5332) Support token-preserving restart of history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776347#comment-13776347 ] Jason Lowe commented on MAPREDUCE-5332: --- The TestMRJobClient and TestUberAM failures are still unrelated and are tracked by MAPREDUCE-5503 and MAPREDUCE-5481, respectively. > Support token-preserving restart of history server > -- > > Key: MAPREDUCE-5332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5332 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-5332-2.patch, MAPREDUCE-5332-3.patch, > MAPREDUCE-5332-4.patch, MAPREDUCE-5332-5.patch, MAPREDUCE-5332-5.patch, > MAPREDUCE-5332-6.patch, MAPREDUCE-5332-7.patch, MAPREDUCE-5332.patch > > > To better support rolling upgrades through a cluster, the history server > needs the ability to restart without losing track of delegation tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5522) Incorrectly expect the array of JobQueueInfo returned by o.a.h.mapred.QueueManager#getJobQueueInfos to have a specific order.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776301#comment-13776301 ] Hudson commented on MAPREDUCE-5522: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1532 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1532/]) MAPREDUCE-5522. Incorrect oreder expected from JobQueueInfo (Jinghui Wang via bobby) (bobby: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525670) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestQueue.java > Incorrectly expect the array of JobQueueInfo returned by > o.a.h.mapred.QueueManager#getJobQueueInfos to have a specific order. > - > > Key: MAPREDUCE-5522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 & IBM Java 1.6 >Reporter: Jinghui Wang >Assignee: Jinghui Wang >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: MAPREDUCE-5522.patch > > > There is a bug in test o.a.h.mapred.TestQueue. The implementation of > getJobQueueInfos in QueueManager uses the keySet of a HashMap to populate the > return value and since there is no guarantee in the ordering of the elements > in the keySet of a Hashmap, this test would fail if the order returned by > getJobQueueInfos is different than what the test is expecting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5522) Incorrectly expect the array of JobQueueInfo returned by o.a.h.mapred.QueueManager#getJobQueueInfos to have a specific order.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776289#comment-13776289 ] Hudson commented on MAPREDUCE-5522: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1558 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1558/]) MAPREDUCE-5522. Incorrect oreder expected from JobQueueInfo (Jinghui Wang via bobby) (bobby: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525670) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestQueue.java > Incorrectly expect the array of JobQueueInfo returned by > o.a.h.mapred.QueueManager#getJobQueueInfos to have a specific order. > - > > Key: MAPREDUCE-5522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 & IBM Java 1.6 >Reporter: Jinghui Wang >Assignee: Jinghui Wang >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: MAPREDUCE-5522.patch > > > There is a bug in test o.a.h.mapred.TestQueue. The implementation of > getJobQueueInfos in QueueManager uses the keySet of a HashMap to populate the > return value and since there is no guarantee in the ordering of the elements > in the keySet of a Hashmap, this test would fail if the order returned by > getJobQueueInfos is different than what the test is expecting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[
https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776261#comment-13776261
]
Avner BenHanoch commented on MAPREDUCE-5329:
Hi Siddharth,
I want to thank you for all your comments and advices.
I have one comment back:
{quote}
Rename "mapreduce.shuffle.provider.plugin.classes" - this is a job specific
configuration. mapreduce.job.shuffle.provider.plugin.classes. Also, this should
be in MRJobConfig and not MRConfig
{quote}
It is true that *shuffle-consumer* is at job level. However, here we are
dealing with *shuffle-provider* which is an aux-service that serves multiple
jobs. Hence, it is not at job level.
Also, if possible, I prefer that we'll stay aligned with the names that were
chosen in MAPREDUCE-4049 (for hadoop-1), there I was requested to fulfil an
opposite comment: ??the value of the constant TT_SHUFFLE_PROVIDER_PLUGIN should
not have 'job' as this plugin is not per job but per TT??
Please let me know if I can ignore this particular comment?
> APPLICATION_INIT is never sent to AuxServices other than the builtin
> ShuffleHandler
> ---
>
> Key: MAPREDUCE-5329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.1.0-beta, 2.0.6-alpha
>Reporter: Avner BenHanoch
> Fix For: trunk
>
> Attachments: MAPREDUCE-5329.patch
>
>
> APPLICATION_INIT is never sent to AuxServices other than the built-in
> ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be
> able to function, because APPLICATION_INIT enables the AuxiliaryService to
> map jobId->userId. This is needed for properly finding the MOFs of a job per
> reducers' requests.
> NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to
> hard-coded expression in hadoop code. The current TaskAttemptImpl.java code
> explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID,
> ...) and ignores any additional AuxiliaryService. As a result, only the
> built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party
> AuxillaryService will never get APPLICATION_INIT events.
> I think a solution can be in one of two ways:
> 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register
> each of them, by calling serviceData.put (…) in loop.
> 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668
> "APPLICATION_STOP is never sent to AuxServices". This means that in case the
> 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux
> Services regardless of the value in event.getServiceID().
> I prefer the 2nd solution. I am welcoming any ideas. I can provide the
> needed patch for any option that people like.
> See [Pluggable Shuffle in Hadoop
> documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5522) Incorrectly expect the array of JobQueueInfo returned by o.a.h.mapred.QueueManager#getJobQueueInfos to have a specific order.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776202#comment-13776202 ] Hudson commented on MAPREDUCE-5522: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #342 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/342/]) MAPREDUCE-5522. Incorrect oreder expected from JobQueueInfo (Jinghui Wang via bobby) (bobby: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525670) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestQueue.java > Incorrectly expect the array of JobQueueInfo returned by > o.a.h.mapred.QueueManager#getJobQueueInfos to have a specific order. > - > > Key: MAPREDUCE-5522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 0.23.7, 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 & IBM Java 1.6 >Reporter: Jinghui Wang >Assignee: Jinghui Wang >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: MAPREDUCE-5522.patch > > > There is a bug in test o.a.h.mapred.TestQueue. The implementation of > getJobQueueInfos in QueueManager uses the keySet of a HashMap to populate the > return value and since there is no guarantee in the ordering of the elements > in the keySet of a Hashmap, this test would fail if the order returned by > getJobQueueInfos is different than what the test is expecting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5503) TestMRJobClient.testJobClient is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776092#comment-13776092
]
Akira AJISAKA commented on MAPREDUCE-5503:
--
[~jianhe], thanks for taking over!
The patch looks good to me.
One comment,
{code}
+runTool(conf, jc, new String[] { "-kill-task", taid.toString() }, out);
+// Killed task attempt_1379653489982_0002_m_00_1
+String answer = new String(out.toByteArray(), "UTF-8");
{code}
The jobID 1379653489982_0002 is different from actual because it contains the
unix time when ResourceManager starts. I think the below comment is more
correct.
{code}
// Killed task attempt__m_00_1
{code}
> TestMRJobClient.testJobClient is failing
>
>
> Key: MAPREDUCE-5503
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5503
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Jason Lowe
>Assignee: Jian He
>Priority: Blocker
> Attachments: MAPREDUCE-5503.patch, MAPREDUCE-5503.patch
>
>
> TestMRJobClient.testJobClient is failing on trunk and causing precommit
> builds to complain:
> {noformat}
> testJobClient(org.apache.hadoop.mapreduce.TestMRJobClient) Time elapsed:
> 26.361 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<1> but was:<0>
> at junit.framework.Assert.fail(Assert.java:50)
> at junit.framework.Assert.failNotEquals(Assert.java:287)
> at junit.framework.Assert.assertEquals(Assert.java:67)
> at junit.framework.Assert.assertEquals(Assert.java:199)
> at junit.framework.Assert.assertEquals(Assert.java:205)
> at
> org.apache.hadoop.mapreduce.TestMRJobClient.testJobList(TestMRJobClient.java:474)
> at
> org.apache.hadoop.mapreduce.TestMRJobClient.testJobClient(TestMRJobClient.java:112)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776065#comment-13776065 ] Siddharth Seth commented on MAPREDUCE-5329: --- Comments on the patch. - Rename "mapreduce.shuffle.provider.plugin.classes" - this is a job specific configuration. mapreduce.job.shuffle.provider.plugin.classes. Also, this should be in MRJobConfig and not MRConfig - Given that this is not going to be used very often, I think it can be left out of mapred-default.xml - and instead documented in MRJobConfig itself. - Would prefer avoiding the fully qualified name (java.util.Collection) - Should the loading walk over the specified ShuffleProvider list, rather than the ones available on the NodeManager - and log providers which are not available. - For the unit test, is the security setup really required - kerberos, acl setup etc. Also, you may want to change the service name to not have a "." in it. YARN-1229 Will get to the next patch quicker. Thanks for the patience. > APPLICATION_INIT is never sent to AuxServices other than the builtin > ShuffleHandler > --- > > Key: MAPREDUCE-5329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.1.0-beta, 2.0.6-alpha >Reporter: Avner BenHanoch > Fix For: trunk > > Attachments: MAPREDUCE-5329.patch > > > APPLICATION_INIT is never sent to AuxServices other than the built-in > ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be > able to function, because APPLICATION_INIT enables the AuxiliaryService to > map jobId->userId. This is needed for properly finding the MOFs of a job per > reducers' requests. > NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to > hard-coded expression in hadoop code. The current TaskAttemptImpl.java code > explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID, > ...) and ignores any additional AuxiliaryService. As a result, only the > built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party > AuxillaryService will never get APPLICATION_INIT events. > I think a solution can be in one of two ways: > 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register > each of them, by calling serviceData.put (…) in loop. > 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668 > "APPLICATION_STOP is never sent to AuxServices". This means that in case the > 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux > Services regardless of the value in event.getServiceID(). > I prefer the 2nd solution. I am welcoming any ideas. I can provide the > needed patch for any option that people like. > See [Pluggable Shuffle in Hadoop > documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776058#comment-13776058
]
Zhijie Shen commented on MAPREDUCE-5529:
[~rkanter], thanks for finding out the remaining binary incompatibility issue.
+1. The patch looks good to me.
> Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
> and branch-2
> --
>
> Key: MAPREDUCE-5529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5529.patch
>
>
> {{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
> {code:java}
> public static String getPartitionFile(JobConf job)
> public static void setPartitionFile(JobConf job, Path p)
> {code}
> In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
> {{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
> methods:
> {code:java}
> public static String getPartitionFile(Configuration conf)
> public static void setPartitionFile(Configuration conf, Path p)
> {code}
> This means that any code that does either of the following:
> {code:java}
> TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
> String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
> {code}
> will not be binary compatible (that is, if compiled against branch-1, it will
> throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated MAPREDUCE-5529:
---
Issue Type: Sub-task (was: Bug)
Parent: MAPREDUCE-5108
> Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1
> and branch-2
> --
>
> Key: MAPREDUCE-5529
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv1, mrv2
>Affects Versions: 2.1.1-beta
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Attachments: MAPREDUCE-5529.patch
>
>
> {{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
> {code:java}
> public static String getPartitionFile(JobConf job)
> public static void setPartitionFile(JobConf job, Path p)
> {code}
> In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of
> {{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar
> methods:
> {code:java}
> public static String getPartitionFile(Configuration conf)
> public static void setPartitionFile(Configuration conf, Path p)
> {code}
> This means that any code that does either of the following:
> {code:java}
> TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
> String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
> {code}
> will not be binary compatible (that is, if compiled against branch-1, it will
> throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
