[jira] [Commented] (MAPREDUCE-4204) Refactor ProcfsBasedProcessTree to make the resource collection object pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276517#comment-13276517 ] Radim Kolar commented on MAPREDUCE-4204: Are you working on trunk version? > Refactor ProcfsBasedProcessTree to make the resource collection object > pluggable > > > Key: MAPREDUCE-4204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4204 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: MAPREDUCE-4204-1.patch, MAPREDUCE-4204.patch > > > Making it a pluggable interface will allow replacing the procfs based > implementation with ones for other platforms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a compile-time dependency on HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-4148: --- Fix Version/s: (was: 2.0.0) 2.0.1 > MapReduce should not have a compile-time dependency on HDFS > --- > > Key: MAPREDUCE-4148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Tom White >Assignee: Tom White > Fix For: 2.0.1 > > Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, > MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, MAPREDUCE-4148.patch, > MAPREDUCE-4148.patch, MAPREDUCE-4148.patch > > > MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token > debug information). We should remove this dependency and MapReduce's > compile-time dependency on HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3543) Mavenize Gridmix.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3543: --- Status: Open (was: Patch Available) > Mavenize Gridmix. > - > > Key: MAPREDUCE-3543 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3543 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Mahadev konar >Assignee: Thomas Graves >Priority: Critical > Attachments: MAPREDUCE-3543v1.patch, MAPREDUCE-3543v1.sh, > MAPREDUCE-3543v2.patch, MAPREDUCE-3543v2.sh, MAPREDUCE-3543v3.patch, > MAPREDUCE-3543v3.sh, MAPREDUCE-3543v3.sh > > > Gridmix codebase still resides in src/contrib and needs to be compiled via > ant. We should move it to maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3543) Mavenize Gridmix.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276484#comment-13276484 ] Robert Joseph Evans commented on MAPREDUCE-3543: Looking at the patch most things look OK, but the patch did not apply correctly, looks like it needs to be upmerged. Also when I tried to build it, after fixing the small issue with the patch to have it apply, I get a build error. Which means the hunks may not have applied completely correctly either. > Mavenize Gridmix. > - > > Key: MAPREDUCE-3543 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3543 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Mahadev konar >Assignee: Thomas Graves >Priority: Critical > Attachments: MAPREDUCE-3543v1.patch, MAPREDUCE-3543v1.sh, > MAPREDUCE-3543v2.patch, MAPREDUCE-3543v2.sh, MAPREDUCE-3543v3.patch, > MAPREDUCE-3543v3.sh, MAPREDUCE-3543v3.sh > > > Gridmix codebase still resides in src/contrib and needs to be compiled via > ant. We should move it to maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4250) hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276464#comment-13276464 ] Hudson commented on MAPREDUCE-4250: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2267 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2267/]) MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh > hadoop-config.sh missing variable exports, causes Yarn jobs to fail with > ClassNotFoundException MRAppMaster > --- > > Key: MAPREDUCE-4250 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4250 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.1 > > Attachments: MAPREDUCE-4250.patch > > > This is the MR side of HADOOP-8393 > If you start a pseudo distributed yarn using "start-yarn.sh" you need to > specify exports for HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, YARN_HOME, > YARN_CONF_DIR, and HADOOP_MAPRED_HOME in hadoop-env.sh (or elsewhere), > otherwise the spawned node manager will be missing > these in it's environment. This is due to start-yarn using yarn-daemons. With > this fix it's possible to start yarn (etc...) with only HADOOP_CONF_DIR > specified in the environment. Took some time to track down this failure, so > seems worthwhile to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4250) hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276444#comment-13276444 ] Hudson commented on MAPREDUCE-4250: --- Integrated in Hadoop-Common-trunk-Commit #2250 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2250/]) MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh > hadoop-config.sh missing variable exports, causes Yarn jobs to fail with > ClassNotFoundException MRAppMaster > --- > > Key: MAPREDUCE-4250 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4250 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.1 > > Attachments: MAPREDUCE-4250.patch > > > This is the MR side of HADOOP-8393 > If you start a pseudo distributed yarn using "start-yarn.sh" you need to > specify exports for HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, YARN_HOME, > YARN_CONF_DIR, and HADOOP_MAPRED_HOME in hadoop-env.sh (or elsewhere), > otherwise the spawned node manager will be missing > these in it's environment. This is due to start-yarn using yarn-daemons. With > this fix it's possible to start yarn (etc...) with only HADOOP_CONF_DIR > specified in the environment. Took some time to track down this failure, so > seems worthwhile to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4250) hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276441#comment-13276441 ] Hudson commented on MAPREDUCE-4250: --- Integrated in Hadoop-Hdfs-trunk-Commit #2324 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2324/]) MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) (Revision 1339000) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339000 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-config.sh > hadoop-config.sh missing variable exports, causes Yarn jobs to fail with > ClassNotFoundException MRAppMaster > --- > > Key: MAPREDUCE-4250 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4250 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.1 > > Attachments: MAPREDUCE-4250.patch > > > This is the MR side of HADOOP-8393 > If you start a pseudo distributed yarn using "start-yarn.sh" you need to > specify exports for HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, YARN_HOME, > YARN_CONF_DIR, and HADOOP_MAPRED_HOME in hadoop-env.sh (or elsewhere), > otherwise the spawned node manager will be missing > these in it's environment. This is due to start-yarn using yarn-daemons. With > this fix it's possible to start yarn (etc...) with only HADOOP_CONF_DIR > specified in the environment. Took some time to track down this failure, so > seems worthwhile to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4163) consistently set the bind address
[
https://issues.apache.org/jira/browse/MAPREDUCE-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276439#comment-13276439
]
Alejandro Abdelnur commented on MAPREDUCE-4163:
---
disregard my prev comment, i didn't have the latest trunk (with MAPREDUCE-4237)
> consistently set the bind address
> -
>
> Key: MAPREDUCE-4163
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4163
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0, 2.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-3659-1.patch, MAPREDUCE-3659-2.patch,
> MAPREDUCE-3659.patch
>
>
> Use {{NetUtils.getConnectAddress}} for determining the bind address used for
> setting a token's service.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4250) hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur resolved MAPREDUCE-4250. --- Resolution: Fixed Fix Version/s: 2.0.1 Hadoop Flags: Reviewed Thanks Patrick. committed to trunk and branch-2. > hadoop-config.sh missing variable exports, causes Yarn jobs to fail with > ClassNotFoundException MRAppMaster > --- > > Key: MAPREDUCE-4250 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4250 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.1 > > Attachments: MAPREDUCE-4250.patch > > > This is the MR side of HADOOP-8393 > If you start a pseudo distributed yarn using "start-yarn.sh" you need to > specify exports for HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, YARN_HOME, > YARN_CONF_DIR, and HADOOP_MAPRED_HOME in hadoop-env.sh (or elsewhere), > otherwise the spawned node manager will be missing > these in it's environment. This is due to start-yarn using yarn-daemons. With > this fix it's possible to start yarn (etc...) with only HADOOP_CONF_DIR > specified in the environment. Took some time to track down this failure, so > seems worthwhile to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3868) Reenable Raid
[ https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276381#comment-13276381 ] Scott Chen commented on MAPREDUCE-3868: --- Weiyang: Thanks for volunteering on this. Looking forward to your patch :) > Reenable Raid > - > > Key: MAPREDUCE-3868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/raid >Reporter: Scott Chen >Assignee: Weiyan Wang > Attachments: MAPREDUCE-3868v1.patch, MAPREDUCE-3868v1.sh > > > Currently Raid is outdated and not compiled. Make it compile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4250) hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276349#comment-13276349 ] Alejandro Abdelnur commented on MAPREDUCE-4250: --- +1 lgtm > hadoop-config.sh missing variable exports, causes Yarn jobs to fail with > ClassNotFoundException MRAppMaster > --- > > Key: MAPREDUCE-4250 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4250 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4250.patch > > > This is the MR side of HADOOP-8393 > If you start a pseudo distributed yarn using "start-yarn.sh" you need to > specify exports for HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, YARN_HOME, > YARN_CONF_DIR, and HADOOP_MAPRED_HOME in hadoop-env.sh (or elsewhere), > otherwise the spawned node manager will be missing > these in it's environment. This is due to start-yarn using yarn-daemons. With > this fix it's possible to start yarn (etc...) with only HADOOP_CONF_DIR > specified in the environment. Took some time to track down this failure, so > seems worthwhile to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4163) consistently set the bind address
[
https://issues.apache.org/jira/browse/MAPREDUCE-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276335#comment-13276335
]
Alejandro Abdelnur commented on MAPREDUCE-4163:
---
What is the rational of the following change in the TestNodeStatusUpdater?
{code}
- try {
-Assert.assertEquals(InetAddress.getLocalHost().getCanonicalHostName()
-+ ":12345", nodeId.toString());
- } catch (UnknownHostException e) {
-Assert.fail(e.getMessage());
- }
+ // NOTE: this really should be checking against the config value
+ Assert.assertEquals("localhost:12345", nodeId.toString());
{code}
This is failing in systems where localhost is not the first hostname in
/etc/hosts for the loopback (i.e. CentOS5 where the first hostname is
localhost.localdomain). Before was working just fine.
> consistently set the bind address
> -
>
> Key: MAPREDUCE-4163
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4163
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0, 2.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-3659-1.patch, MAPREDUCE-3659-2.patch,
> MAPREDUCE-3659.patch
>
>
> Use {{NetUtils.getConnectAddress}} for determining the bind address used for
> setting a token's service.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4260) Investigate use of JobObject to spawn tasks on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-4260: -- Description: Currently, the Windows version spawns the task as a normal cmd shell from which other downstream exe's are spawned. However, this is not bullet proof because if an intermediate process exits before its child exits, then the parent child process tree relationship cannot be constructed. Windows has a concept of JobObject that is similar to the setsid behavior used in Linux. The initial spawned task could be launched within its JobObject. Thereafter, process termination, memory management etc could be operated on the JobObject. Assignee: Bikas Saha > Investigate use of JobObject to spawn tasks on Windows > -- > > Key: MAPREDUCE-4260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4260 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Bikas Saha >Assignee: Bikas Saha > > Currently, the Windows version spawns the task as a normal cmd shell from > which other downstream exe's are spawned. However, this is not bullet proof > because if an intermediate process exits before its child exits, then the > parent child process tree relationship cannot be constructed. Windows has a > concept of JobObject that is similar to the setsid behavior used in Linux. > The initial spawned task could be launched within its JobObject. Thereafter, > process termination, memory management etc could be operated on the JobObject. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (MAPREDUCE-4260) Investigate use of JobObject to spawn tasks on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha moved HADOOP-8401 to MAPREDUCE-4260: --- Affects Version/s: (was: 1.0.0) 1.0.0 Key: MAPREDUCE-4260 (was: HADOOP-8401) Project: Hadoop Map/Reduce (was: Hadoop Common) > Investigate use of JobObject to spawn tasks on Windows > -- > > Key: MAPREDUCE-4260 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4260 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Bikas Saha > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3350) Per-app RM page should have the list of application-attempts like on the app JHS page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276242#comment-13276242 ] Jonathan Eagles commented on MAPREDUCE-3350: Need to add web services to the code before it is ready. > Per-app RM page should have the list of application-attempts like on the app > JHS page > - > > Key: MAPREDUCE-3350 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3350 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, webapps >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.24.0 > > Attachments: MAPREDUCE-3350.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3350) Per-app RM page should have the list of application-attempts like on the app JHS page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3350: --- Attachment: MAPREDUCE-3350.patch > Per-app RM page should have the list of application-attempts like on the app > JHS page > - > > Key: MAPREDUCE-3350 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3350 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, webapps >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.24.0 > > Attachments: MAPREDUCE-3350.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276225#comment-13276225 ] Matei Zaharia commented on MAPREDUCE-3451: -- Having a single very feature-rich scheduler is certainly useful, but that's a separate discussion. My point was just that some people want compatibility with the MR1 Fair Scheduler as a feature -- not having that is an obstacle to their adopting MR2. I also think it would be hard to design a single scheduler that will support all the features people might want in the future, for the same reason that it was hard to get different development groups to build a single scheduler in MR1. You just can't anticipate all the needs, and how various features will interact -- for example, Yahoo! had to remove preemption from the capacity scheduler because it complicated the design. We are also planning to add some features to the Fair Scheduler ourselves, such as multi-resource fairness, that might require major changes to other schedulers even if we were to wait on them adopting the existing features. > Port Fair Scheduler to MR2 > -- > > Key: MAPREDUCE-3451 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, scheduler >Reporter: Patrick Wendell >Assignee: Patrick Wendell > Attachments: MAPREDUCE-3451.v1.patch.txt, > MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, > MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch > > > The Fair Scheduler is in widespread use today in MR1 clusters, but not yet > ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and > will be updated to include design considerations and progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276203#comment-13276203 ] Tom White commented on MAPREDUCE-3451: -- Matei - I'm not against this going in (although we should put it in a separate module), but I thought it would be a good time to have a discussion about supporting a single main scheduler in YARN. If we could combine features from the various schedulers in MR1 to YARN then we would have less code to support and test, and it would be less confusing for users. What do you think? Do you agree that this would be a useful goal? > Port Fair Scheduler to MR2 > -- > > Key: MAPREDUCE-3451 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, scheduler >Reporter: Patrick Wendell >Assignee: Patrick Wendell > Attachments: MAPREDUCE-3451.v1.patch.txt, > MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, > MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch > > > The Fair Scheduler is in widespread use today in MR1 clusters, but not yet > ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and > will be updated to include design considerations and progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4238) mavenize data_join
[ https://issues.apache.org/jira/browse/MAPREDUCE-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276165#comment-13276165 ] Hudson commented on MAPREDUCE-4238: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2265 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2265/]) MAPREDUCE-4238. mavenize data_join. (tgraves) (Revision 1338835) Result = ABORTED tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338835 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/README.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/SampleTaggedMapOutput.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/test/org * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main/java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main/java/org/apache/hadoop/contrib/utils/join/JobBase.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleTaggedMapOutput.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/TestDataJoin.java * /hadoop/common/trunk/hadoop-tools/hadoop-tools-dist/pom.xml * /hadoop/common/trunk/hadoop-tools/pom.xml > mavenize data_join > -- > > Key: MAPREDUCE-4238 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4238 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4238.patch, MAPREDUCE-4238.patch, > MAPREDUCE-4238.sh > > > mavenize the contrib data_join package -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4238) mavenize data_join
[ https://issues.apache.org/jira/browse/MAPREDUCE-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4238: - Resolution: Fixed Fix Version/s: 0.23.3 Status: Resolved (was: Patch Available) I committed this to trunk, branch-2, and branch-0.23 > mavenize data_join > -- > > Key: MAPREDUCE-4238 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4238 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4238.patch, MAPREDUCE-4238.patch, > MAPREDUCE-4238.sh > > > mavenize the contrib data_join package -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4238) mavenize data_join
[ https://issues.apache.org/jira/browse/MAPREDUCE-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276129#comment-13276129 ] Hudson commented on MAPREDUCE-4238: --- Integrated in Hadoop-Common-trunk-Commit #2248 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2248/]) MAPREDUCE-4238. mavenize data_join. (tgraves) (Revision 1338835) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338835 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/README.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/SampleTaggedMapOutput.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/test/org * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main/java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main/java/org/apache/hadoop/contrib/utils/join/JobBase.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleTaggedMapOutput.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/TestDataJoin.java * /hadoop/common/trunk/hadoop-tools/hadoop-tools-dist/pom.xml * /hadoop/common/trunk/hadoop-tools/pom.xml > mavenize data_join > -- > > Key: MAPREDUCE-4238 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4238 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: MAPREDUCE-4238.patch, MAPREDUCE-4238.patch, > MAPREDUCE-4238.sh > > > mavenize the contrib data_join package -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4238) mavenize data_join
[ https://issues.apache.org/jira/browse/MAPREDUCE-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276122#comment-13276122 ] Hudson commented on MAPREDUCE-4238: --- Integrated in Hadoop-Hdfs-trunk-Commit #2322 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2322/]) MAPREDUCE-4238. mavenize data_join. (tgraves) (Revision 1338835) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1338835 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/README.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/SampleTaggedMapOutput.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/data_join/src/test/org * /hadoop/common/trunk/hadoop-project/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main/java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/main/java/org/apache/hadoop/contrib/utils/join/JobBase.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleTaggedMapOutput.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/TestDataJoin.java * /hadoop/common/trunk/hadoop-tools/hadoop-tools-dist/pom.xml * /hadoop/common/trunk/hadoop-tools/pom.xml > mavenize data_join > -- > > Key: MAPREDUCE-4238 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4238 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: MAPREDUCE-4238.patch, MAPREDUCE-4238.patch, > MAPREDUCE-4238.sh > > > mavenize the contrib data_join package -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4146) Support limits on task status string length and number of block locations in branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276112#comment-13276112 ] Hadoop QA commented on MAPREDUCE-4146: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527473/MAPREDUCE-4146_rev2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2388//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2388//console This message is automatically generated. > Support limits on task status string length and number of block locations in > branch-2 > - > > Key: MAPREDUCE-4146 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4146 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-4146.patch, MAPREDUCE-4146_rev2.patch > > > This brings MAPREDUCE-1943 to branch-2. Counter limits were introduced in > MAPREDUCE-901. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4152) map task left hanging after AM dies trying to connect to RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-4152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276079#comment-13276079 ] Thomas Graves commented on MAPREDUCE-4152: -- Vinod, comments? > map task left hanging after AM dies trying to connect to RM > --- > > Key: MAPREDUCE-4152 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4152 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4152.patch, MAPREDUCE-4152.patch > > > We had an instance where the RM went down for more then an hour. The > application master exited with "Could not contact RM after 36 > milliseconds" > 2012-04-11 10:43:36,040 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: > job_1333003059741_15999Job Transitioned from RUNNING to ERROR -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4146) Support limits on task status string length and number of block locations in branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-4146: Attachment: MAPREDUCE-4146_rev2.patch > Support limits on task status string length and number of block locations in > branch-2 > - > > Key: MAPREDUCE-4146 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4146 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-4146.patch, MAPREDUCE-4146_rev2.patch > > > This brings MAPREDUCE-1943 to branch-2. Counter limits were introduced in > MAPREDUCE-901. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4146) Support limits on task status string length and number of block locations in branch-2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-4146: Status: Patch Available (was: Open) > Support limits on task status string length and number of block locations in > branch-2 > - > > Key: MAPREDUCE-4146 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4146 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-4146.patch, MAPREDUCE-4146_rev2.patch > > > This brings MAPREDUCE-1943 to branch-2. Counter limits were introduced in > MAPREDUCE-901. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4250) hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276025#comment-13276025 ] Ahmed Radwan commented on MAPREDUCE-4250: - +1 for the patch. Thanks Patrick! > hadoop-config.sh missing variable exports, causes Yarn jobs to fail with > ClassNotFoundException MRAppMaster > --- > > Key: MAPREDUCE-4250 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4250 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Attachments: MAPREDUCE-4250.patch > > > This is the MR side of HADOOP-8393 > If you start a pseudo distributed yarn using "start-yarn.sh" you need to > specify exports for HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, YARN_HOME, > YARN_CONF_DIR, and HADOOP_MAPRED_HOME in hadoop-env.sh (or elsewhere), > otherwise the spawned node manager will be missing > these in it's environment. This is due to start-yarn using yarn-daemons. With > this fix it's possible to start yarn (etc...) with only HADOOP_CONF_DIR > specified in the environment. Took some time to track down this failure, so > seems worthwhile to fix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4259) TestDatanodeBlockScanner and TestReplication fail intermittently on Windows
Bikas Saha created MAPREDUCE-4259: - Summary: TestDatanodeBlockScanner and TestReplication fail intermittently on Windows Key: MAPREDUCE-4259 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4259 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.0.0 Reporter: Bikas Saha Assignee: Bikas Saha The tests change the block length to corrupt the data block. If the block file is opened by the datanode then the test can concurrently modify it on Linux but such concurrent modification is not allowed by the default permissions on Windows. Since this is more of a test issue, the fix would be to have the tests make sure that the block is not open concurrently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4252) MR2 job never completes with 1 pending task
[ https://issues.apache.org/jira/browse/MAPREDUCE-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13275995#comment-13275995 ] Bikas Saha commented on MAPREDUCE-4252: --- @Sid ,thanks for clarifying. @Tom, Let me clarify a bit. When the speculative attempt succeeded, then AttemptSucceededTransition would have KILLED the other running attempts. In this case, the other attempt was not KILLED but it FAILED on its own. > MR2 job never completes with 1 pending task > --- > > Key: MAPREDUCE-4252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4252 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-4252.patch, MAPREDUCE-4252.patch, MapReduce.png > > > This was found by ATM: > bq. I ran a teragen with 1000 map tasks. Many task attempts failed, but after > 999 of the tasks had completed, the job is now sitting forever with 1 task > "pending". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4252) MR2 job never completes with 1 pending task
[ https://issues.apache.org/jira/browse/MAPREDUCE-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13275980#comment-13275980 ] Siddharth Seth commented on MAPREDUCE-4252: --- bq. Do you mean decrement numberUncompletedAttempts when a RUNNING attempt fails/killed after the task is in SUCCEEDED state? Normally they would trigger their own Failed/Killed transition that takes care of the decrement. After a task goes to SUCCEEDED, FAILED/KILLED attempts are ignored. 1. attemp1 starts 2. speculative attempt starts 3. attempt 1 completes - Task moves to SUCCEEDED state 4. speculative attempt is KILLED 5. T_ATTEMPT_KILLED is ignored. 6. attemp1 1 fails with TOO_MANY_FETCH_FAILURES The job will effectively hang, since a new task attempt isn't started. @Tom, this is another case which leads to the same situation of a stuck job. I can open a separate jira if you plan to commit this patch as is. > MR2 job never completes with 1 pending task > --- > > Key: MAPREDUCE-4252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4252 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-4252.patch, MAPREDUCE-4252.patch, MapReduce.png > > > This was found by ATM: > bq. I ran a teragen with 1000 map tasks. Many task attempts failed, but after > 999 of the tasks had completed, the job is now sitting forever with 1 task > "pending". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
