[jira] [Commented] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118690#comment-13118690 ] Chris Riccomini commented on MAPREDUCE-3068: Any feedback on this? > Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers > --- > > Key: MAPREDUCE-3068 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Chris Riccomini > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff > > > This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should > all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118686#comment-13118686 ] Hudson commented on MAPREDUCE-2889: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1018 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1018/]) MAPREDUCE-2889. Added documentation for writing new YARN applications. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177920 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/src/site/apt/SingleCluster.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/src/site/apt/index.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/src/site/site.xml > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2889.patch, MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118682#comment-13118682 ] Subroto Sanyal commented on MAPREDUCE-3131: --- hi Prashant, The existing documentation: 1) ~hadoop-trunk/BUILDING.txt 2) ~hadoop-trunk/hadoop-mapreduce-project/INSTALL clearly tells a user: a) How to build the project b) How to get the distributable? c) How to run the Yarn Cluster? IMO the existing documentation and scripts are enough. > Docs and Scripts for setting up single node MRV2 cluster. > -- > > Key: MAPREDUCE-3131 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Reporter: Prashant Sharma > Fix For: 0.23.0 > > Original Estimate: 168h > Remaining Estimate: 168h > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118684#comment-13118684 ] Hadoop QA commented on MAPREDUCE-2889: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497267/MAPREDUCE-2889.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 69 javac compiler warnings (more than the trunk's current 66 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/920//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/920//console This message is automatically generated. > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2889.patch, MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118680#comment-13118680 ] Hudson commented on MAPREDUCE-2889: --- Integrated in Hadoop-Common-trunk-Commit #998 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/998/]) MAPREDUCE-2889. Added documentation for writing new YARN applications. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177920 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/src/site/apt/SingleCluster.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/src/site/apt/index.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/src/site/site.xml > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2889.patch, MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118678#comment-13118678 ] Hudson commented on MAPREDUCE-2889: --- Integrated in Hadoop-Hdfs-trunk-Commit #1076 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1076/]) MAPREDUCE-2889. Added documentation for writing new YARN applications. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177920 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/site.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/src/site/apt/SingleCluster.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/src/site/apt/index.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/src/site/site.xml > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2889.patch, MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2889: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2889.patch, MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2889: - Attachment: MAPREDUCE-2889.patch Looks great Hitesh, thanks! I just did minor edits, ensured 80-char limit etc. > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2889.patch, MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118654#comment-13118654 ] Hitesh Shah commented on MAPREDUCE-2889: Additional warnings due to apache-rat-plugin warnings on a new module. > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118628#comment-13118628 ] Hadoop QA commented on MAPREDUCE-2889: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497257/MR-2889.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 69 javac compiler warnings (more than the trunk's current 66 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/919//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/919//console This message is automatically generated. > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2889: --- Attachment: MR-2889.2.patch Moved docs under a new hadoop-yarn-site module. Incorporated faq back into main page. > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2889: --- Status: Patch Available (was: Open) > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2889: --- Status: Open (was: Patch Available) > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-2889.1.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118612#comment-13118612 ] Hudson commented on MAPREDUCE-2779: --- Integrated in Hadoop-Mapreduce-22-branch #79 (See [https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/79/]) MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file. Contributed by Ming Ma. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177787 Files : * /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt * /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapreduce/split/JobSplitWriter.java > JobSplitWriter.java can't handle large job.split file > - > > Key: MAPREDUCE-2779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.20.205.0, 0.22.0, 0.23.0 >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 0.22.0, 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2779-0.22.patch, MAPREDUCE-2779-trunk.patch, > MAPREDUCE-2779-trunk.patch > > > We use cascading MultiInputFormat. MultiInputFormat sometimes generates big > job.split used internally by hadoop, sometimes it can go beyond 2GB. > In JobSplitWriter.java, the function that generates such file uses 32bit > signed integer to compute offset into job.split. > writeNewSplits > ... > int prevCount = out.size(); > ... > int currCount = out.size(); > writeOldSplits > ... > long offset = out.size(); > ... > int currLen = out.size(); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3113) the scripts yarn-daemon.sh and yarn are not working properly
[
https://issues.apache.org/jira/browse/MAPREDUCE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118595#comment-13118595
]
Hudson commented on MAPREDUCE-3113:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1016 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1016/])
MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root of
the install properly. Contributed by Xie Xianshan.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177874
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
> the scripts yarn-daemon.sh and yarn are not working properly
>
>
> Key: MAPREDUCE-3113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3113
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: XieXianshan
>Assignee: XieXianshan
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3113.patch
>
>
> When we execute them on any path but $YARN_HOME with bash -x option,it is
> giving the error as follows:
> (Of course we should set the path variable of that scritps into the .bashrc
> or profile in advance)
> {code}
> /usr/share/hadoop/hadoop-mapreduce-0.24.0-SNAPSHOT/bin/yarn: line 55:
> /usr/share/hadoop/yarn-config.sh: No such file or directory
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2266) JvmManager sleeps between SIGTERM and SIGKILL while holding many TT locks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118593#comment-13118593 ] Todd Lipcon commented on MAPREDUCE-2266: There's no patch, since this was dealt with by MAPREDUCE-2178 where JvmManager was substantially rewritten. I can't contribute time to the 0.22 release, since I'm concentrating on 0.23 and 0.20.20x release lines. > JvmManager sleeps between SIGTERM and SIGKILL while holding many TT locks > - > > Key: MAPREDUCE-2266 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2266 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > Fix For: 0.22.0 > > > Between sending a task SIGTERM and SIGKILL, the JvmManager will sleep for > sleepTimeBeforeSigKill millis. But in many call heirarchies this is done > while holding important locks like the TT lock and the JvmManagerForType > lock. With the default 5 second sleep, this prevents other tasks from getting > scheduled and reduces scheduling throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3113) the scripts yarn-daemon.sh and yarn are not working properly
[
https://issues.apache.org/jira/browse/MAPREDUCE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118584#comment-13118584
]
Hudson commented on MAPREDUCE-3113:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1074 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1074/])
MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root of
the install properly. Contributed by Xie Xianshan.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177874
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
> the scripts yarn-daemon.sh and yarn are not working properly
>
>
> Key: MAPREDUCE-3113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3113
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: XieXianshan
>Assignee: XieXianshan
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3113.patch
>
>
> When we execute them on any path but $YARN_HOME with bash -x option,it is
> giving the error as follows:
> (Of course we should set the path variable of that scritps into the .bashrc
> or profile in advance)
> {code}
> /usr/share/hadoop/hadoop-mapreduce-0.24.0-SNAPSHOT/bin/yarn: line 55:
> /usr/share/hadoop/yarn-config.sh: No such file or directory
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3113) the scripts yarn-daemon.sh and yarn are not working properly
[
https://issues.apache.org/jira/browse/MAPREDUCE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118583#comment-13118583
]
Hudson commented on MAPREDUCE-3113:
---
Integrated in Hadoop-Common-trunk-Commit #996 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/996/])
MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root of
the install properly. Contributed by Xie Xianshan.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177874
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
> the scripts yarn-daemon.sh and yarn are not working properly
>
>
> Key: MAPREDUCE-3113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3113
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: XieXianshan
>Assignee: XieXianshan
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3113.patch
>
>
> When we execute them on any path but $YARN_HOME with bash -x option,it is
> giving the error as follows:
> (Of course we should set the path variable of that scritps into the .bashrc
> or profile in advance)
> {code}
> /usr/share/hadoop/hadoop-mapreduce-0.24.0-SNAPSHOT/bin/yarn: line 55:
> /usr/share/hadoop/yarn-config.sh: No such file or directory
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3113) the scripts yarn-daemon.sh and yarn are not working properly
[
https://issues.apache.org/jira/browse/MAPREDUCE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3113:
-
Resolution: Fixed
Fix Version/s: (was: 0.24.0)
0.23.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks XieXianshan!
> the scripts yarn-daemon.sh and yarn are not working properly
>
>
> Key: MAPREDUCE-3113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3113
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: XieXianshan
>Assignee: XieXianshan
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3113.patch
>
>
> When we execute them on any path but $YARN_HOME with bash -x option,it is
> giving the error as follows:
> (Of course we should set the path variable of that scritps into the .bashrc
> or profile in advance)
> {code}
> /usr/share/hadoop/hadoop-mapreduce-0.24.0-SNAPSHOT/bin/yarn: line 55:
> /usr/share/hadoop/yarn-config.sh: No such file or directory
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2266) JvmManager sleeps between SIGTERM and SIGKILL while holding many TT locks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118576#comment-13118576 ] Konstantin Shvachko commented on MAPREDUCE-2266: Todd. What do I include? If there was a patch I would be happy to. > JvmManager sleeps between SIGTERM and SIGKILL while holding many TT locks > - > > Key: MAPREDUCE-2266 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2266 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > Fix For: 0.22.0 > > > Between sending a task SIGTERM and SIGKILL, the JvmManager will sleep for > sleepTimeBeforeSigKill millis. But in many call heirarchies this is done > while holding important locks like the TT lock and the JvmManagerForType > lock. With the default 5 second sleep, this prevents other tasks from getting > scheduled and reduces scheduling throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118554#comment-13118554 ] Hudson commented on MAPREDUCE-2719: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1015 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1015/]) MAPREDUCE-2719. Add a simple, DistributedShell, application to illustrate alternate frameworks on YARN. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177864 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/DSConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Iss
[jira] [Commented] (MAPREDUCE-3113) the scripts yarn-daemon.sh and yarn are not working properly
[
https://issues.apache.org/jira/browse/MAPREDUCE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118537#comment-13118537
]
Eric Yang commented on MAPREDUCE-3113:
--
+1 looks good.
> the scripts yarn-daemon.sh and yarn are not working properly
>
>
> Key: MAPREDUCE-3113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3113
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: XieXianshan
>Assignee: XieXianshan
>Priority: Minor
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3113.patch
>
>
> When we execute them on any path but $YARN_HOME with bash -x option,it is
> giving the error as follows:
> (Of course we should set the path variable of that scritps into the .bashrc
> or profile in advance)
> {code}
> /usr/share/hadoop/hadoop-mapreduce-0.24.0-SNAPSHOT/bin/yarn: line 55:
> /usr/share/hadoop/yarn-config.sh: No such file or directory
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118536#comment-13118536 ] Hudson commented on MAPREDUCE-3050: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1014 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1014/]) MAPREDUCE-3050. Add ability to get resource usage information for applications and nodes. Contributed by Robert Evans. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177859 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNodeReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118528#comment-13118528 ] Hudson commented on MAPREDUCE-2719: --- Integrated in Hadoop-Common-trunk-Commit #995 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/995/]) MAPREDUCE-2719. Add a simple, DistributedShell, application to illustrate alternate frameworks on YARN. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177864 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/DSConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type:
[jira] [Commented] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118526#comment-13118526 ] Hudson commented on MAPREDUCE-2719: --- Integrated in Hadoop-Hdfs-trunk-Commit #1073 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1073/]) MAPREDUCE-2719. Add a simple, DistributedShell, application to illustrate alternate frameworks on YARN. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177864 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/DSConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: N
[jira] [Updated] (MAPREDUCE-3013) Remove YarnConfiguration.YARN_SECURITY_INFO
[ https://issues.apache.org/jira/browse/MAPREDUCE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3013: - Status: Open (was: Patch Available) Sorry, you need to rebase this. > Remove YarnConfiguration.YARN_SECURITY_INFO > --- > > Key: MAPREDUCE-3013 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3013 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3013-20110930.txt > > > We don't need this anymore since RPC client uses SecurityUtil to pick it up > via going through the providers for SecurityInfo interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2719: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-2179.1.patch, MR-2179.2.patch, MR-2179.3.patch, > MR-2719.4.post3098.patch, MR-2719.5.patch, MR-2719.6.patch, mr-2719.wip.patch > > > With nextgen hadoop (mrv2), it is simple to write non-MR applications. Write > an AplicationMaster (also corresponding simple client), to submit and run a > shell command application in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118517#comment-13118517 ] Hudson commented on MAPREDUCE-3050: --- Integrated in Hadoop-Common-trunk-Commit #994 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/994/]) MAPREDUCE-3050. Add ability to get resource usage information for applications and nodes. Contributed by Robert Evans. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177859 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNodeReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118515#comment-13118515 ] Hudson commented on MAPREDUCE-3050: --- Integrated in Hadoop-Hdfs-trunk-Commit #1072 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1072/]) MAPREDUCE-3050. Add ability to get resource usage information for applications and nodes. Contributed by Robert Evans. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177859 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerAppReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNodeReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3050: - Resolution: Fixed Fix Version/s: (was: 0.24.0) Target Version/s: 0.23.0 (was: 0.23.0, 0.24.0) Status: Resolved (was: Patch Available) I just committed this. Thanks Robert! > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3056) Jobs are failing when those are submitted by other users
[ https://issues.apache.org/jira/browse/MAPREDUCE-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118503#comment-13118503 ] Arun C Murthy commented on MAPREDUCE-3056: -- Also, don't get user name for config. Always use UGI.getCurrentUser. > Jobs are failing when those are submitted by other users > > > Key: MAPREDUCE-3056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Devaraj K >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3056.patch > > > MR cluster is started by the user 'root'. If any other users other than > 'root' submit a job, it is failing always. > Find the conatiner logs in the comments section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118502#comment-13118502 ] Hadoop QA commented on MAPREDUCE-2889: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497227/MR-2889.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/918//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/918//console This message is automatically generated. > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-2889.1.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3056) Jobs are failing when those are submitted by other users
[ https://issues.apache.org/jira/browse/MAPREDUCE-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3056: - Status: Open (was: Patch Available) Agree with atm. Please provide a test case. Thanks! > Jobs are failing when those are submitted by other users > > > Key: MAPREDUCE-3056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Devaraj K >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3056.patch > > > MR cluster is started by the user 'root'. If any other users other than > 'root' submit a job, it is failing always. > Find the conatiner logs in the comments section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2889: --- Status: Patch Available (was: Open) > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-2889.1.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2889: --- Attachment: MR-2889.1.patch First cut at a doc for writing yarn applications. > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-2889.1.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118483#comment-13118483 ] Hadoop QA commented on MAPREDUCE-2719: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497219/MR-2719.6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 66 javac compiler warnings (more than the trunk's current 60 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/917//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/917//console This message is automatically generated. > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-2179.1.patch, MR-2179.2.patch, MR-2179.3.patch, > MR-2719.4.post3098.patch, MR-2719.5.patch, MR-2719.6.patch, mr-2719.wip.patch > > > With nextgen hadoop (mrv2), it is simple to write non-MR applications. Write > an AplicationMaster (also corresponding simple client), to submit and run a > shell command application in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated MAPREDUCE-2764:
Status: Open (was: Patch Available)
> Fix renewal of dfs delegation tokens
>
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch,
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch,
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.
> The problem is the JT's delegation token renewal code is built on brittle
> assumptions. The token's service field contains only the "ip:port" pair.
> The renewal process assumes that the scheme must be hdfs. If that fails due
> to a {{VersionMismatchException}}, it tries https based on another assumption
> that it must be hftp if it's not hdfs. A number of other exceptions, most
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since
> it won't fallback to https.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens
[
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated MAPREDUCE-2764:
Status: Patch Available (was: Open)
> Fix renewal of dfs delegation tokens
>
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch,
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch,
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.
> The problem is the JT's delegation token renewal code is built on brittle
> assumptions. The token's service field contains only the "ip:port" pair.
> The renewal process assumes that the scheme must be hdfs. If that fails due
> to a {{VersionMismatchException}}, it tries https based on another assumption
> that it must be hftp if it's not hdfs. A number of other exceptions, most
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since
> it won't fallback to https.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2719: --- Status: Patch Available (was: Open) > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-2179.1.patch, MR-2179.2.patch, MR-2179.3.patch, > MR-2719.4.post3098.patch, MR-2719.5.patch, MR-2719.6.patch, mr-2719.wip.patch > > > With nextgen hadoop (mrv2), it is simple to write non-MR applications. Write > an AplicationMaster (also corresponding simple client), to submit and run a > shell command application in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2719: --- Attachment: MR-2719.6.patch Address javadoc warning issue. > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-2179.1.patch, MR-2179.2.patch, MR-2179.3.patch, > MR-2719.4.post3098.patch, MR-2719.5.patch, MR-2719.6.patch, mr-2719.wip.patch > > > With nextgen hadoop (mrv2), it is simple to write non-MR applications. Write > an AplicationMaster (also corresponding simple client), to submit and run a > shell command application in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118432#comment-13118432 ] Hadoop QA commented on MAPREDUCE-3050: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497212/MR-3050.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/916//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/916//console This message is automatically generated. > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3132) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118430#comment-13118430 ] Hadoop QA commented on MAPREDUCE-3132: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497214/MAPREDUCE-3132.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/915//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/915//console This message is automatically generated. > Running a set of methods in a Single Test Class > --- > > Key: MAPREDUCE-3132 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3132 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch > > > Instead of running every test method in a class, limit to specific testing > methods as describe in the link below. > http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html > Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2719: --- Status: Open (was: Patch Available) > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-2179.1.patch, MR-2179.2.patch, MR-2179.3.patch, > MR-2719.4.post3098.patch, MR-2719.5.patch, mr-2719.wip.patch > > > With nextgen hadoop (mrv2), it is simple to write non-MR applications. Write > an AplicationMaster (also corresponding simple client), to submit and run a > shell command application in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118406#comment-13118406 ] Hadoop QA commented on MAPREDUCE-2719: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497203/MR-2719.5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 66 javac compiler warnings (more than the trunk's current 60 warnings). +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/913//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/913//console This message is automatically generated. > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-2179.1.patch, MR-2179.2.patch, MR-2179.3.patch, > MR-2719.4.post3098.patch, MR-2719.5.patch, mr-2719.wip.patch > > > With nextgen hadoop (mrv2), it is simple to write non-MR applications. Write > an AplicationMaster (also corresponding simple client), to submit and run a > shell command application in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3132) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3132: --- Attachment: MAPREDUCE-3132.patch Trying without cross project patch > Running a set of methods in a Single Test Class > --- > > Key: MAPREDUCE-3132 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3132 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch > > > Instead of running every test method in a class, limit to specific testing > methods as describe in the link below. > http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html > Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118400#comment-13118400 ] Hadoop QA commented on MAPREDUCE-2988: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497206/MR-2988.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/912//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/912//console This message is automatically generated. > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3050: --- Attachment: MR-3050.txt Refactored SchedulerNodeReport and SchedulerAppReport to not be a part of ScheduelrNode and SchedulerApp respectively. > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3050: --- Status: Patch Available (was: Open) > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3132) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118386#comment-13118386 ] Jonathan Eagles commented on MAPREDUCE-3132: pom.xml changes only in this patch so no tests cases added manual test was performed to verify new features are working on standard tests as well as classes with @Before, @After, @BeforeClass, and @AfterClass annotations. verified .m2 cache that only maven-surefire-plugin version 2.9 is downloaded into the cache from a clean cache > Running a set of methods in a Single Test Class > --- > > Key: MAPREDUCE-3132 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3132 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-3132.patch > > > Instead of running every test method in a class, limit to specific testing > methods as describe in the link below. > http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html > Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3132) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118382#comment-13118382 ] Hadoop QA commented on MAPREDUCE-3132: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497208/MAPREDUCE-3132.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/914//console This message is automatically generated. > Running a set of methods in a Single Test Class > --- > > Key: MAPREDUCE-3132 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3132 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-3132.patch > > > Instead of running every test method in a class, limit to specific testing > methods as describe in the link below. > http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html > Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3132) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118376#comment-13118376 ] Jonathan Eagles commented on MAPREDUCE-3132: -1 overall. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. > Running a set of methods in a Single Test Class > --- > > Key: MAPREDUCE-3132 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3132 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-3132.patch > > > Instead of running every test method in a class, limit to specific testing > methods as describe in the link below. > http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html > Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118371#comment-13118371 ] Hudson commented on MAPREDUCE-2779: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1013 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1013/]) MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file. Contributed by Ming Ma. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=119 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/JobSplitWriter.java > JobSplitWriter.java can't handle large job.split file > - > > Key: MAPREDUCE-2779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.20.205.0, 0.22.0, 0.23.0 >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 0.22.0, 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2779-0.22.patch, MAPREDUCE-2779-trunk.patch, > MAPREDUCE-2779-trunk.patch > > > We use cascading MultiInputFormat. MultiInputFormat sometimes generates big > job.split used internally by hadoop, sometimes it can go beyond 2GB. > In JobSplitWriter.java, the function that generates such file uses 32bit > signed integer to compute offset into job.split. > writeNewSplits > ... > int prevCount = out.size(); > ... > int currCount = out.size(); > writeOldSplits > ... > long offset = out.size(); > ... > int currLen = out.size(); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
e.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/event/RMAppAttemptUnregistrationEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestASMStateMachine.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestApplicationMasterExpiry.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java > Report Application status as well as ApplicationMaster status in > GetApplicationReportResponse > -- > > Key: MAPREDUCE-3098 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3098-20110930.txt, MR-3098.1.patch, > MR-3098.2.patch, MR-3098.3.patch, MR-3098.4.patch, MR-3098.5.patch > > > Currently, an application report received by the client from the RM/ASM for a > given application returns the status of the application master. It does not > return the status of the application i.e. whether that particular job > succeeded or failed. > The AM status would be one of FINISHED (SUCCEEDED should be renamed to > FINISHED as AM state does not indicate overall success/failure), FAILED or &g
[jira] [Commented] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118369#comment-13118369 ] Hudson commented on MAPREDUCE-2791: --- Integrated in Hadoop-Mapreduce-trunk #846 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/846/]) MAPREDUCE-2791. Added missing info on 'job -status' output. Contributed by Devaraj K. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177487 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRBuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java > [MR-279] Missing/incorrect info on job -status CLI > --- > > Key: MAPREDUCE-2791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2791-1.patch, MAPREDUCE-2791-1.patch, > MAPREDUCE-2791-2.patch, MAPREDUCE-2791-3.patch, MAPREDUCE-2791-4.patch, > MAPREDUCE-2791-5.patch, MAPREDUCE-2791.patch, MAPREDUCE-2791.patch, mvn > install.log > > > There are a couple of details missing/incorrect on the job -status command > line output for completed jobs: > 1. Incorrect job file > 2. map() completion is always 0 > 3. reduce() completion is always set to 0 > 4. history URL is empty > 5. Missing launched map tasks > 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118361#comment-13118361 ] Hudson commented on MAPREDUCE-2779: --- Integrated in Hadoop-Mapreduce-trunk #846 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/846/]) MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file. Contributed by Ming Ma. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=119 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/JobSplitWriter.java > JobSplitWriter.java can't handle large job.split file > - > > Key: MAPREDUCE-2779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.20.205.0, 0.22.0, 0.23.0 >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 0.22.0, 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2779-0.22.patch, MAPREDUCE-2779-trunk.patch, > MAPREDUCE-2779-trunk.patch > > > We use cascading MultiInputFormat. MultiInputFormat sometimes generates big > job.split used internally by hadoop, sometimes it can go beyond 2GB. > In JobSplitWriter.java, the function that generates such file uses 32bit > signed integer to compute offset into job.split. > writeNewSplits > ... > int prevCount = out.size(); > ... > int currCount = out.size(); > writeOldSplits > ... > long offset = out.size(); > ... > int currLen = out.size(); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
mmon/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/event/RMAppAttemptUnregistrationEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestASMStateMachine.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestApplicationMasterExpiry.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java > Report Application status as well as ApplicationMaster status in > GetApplicationReportResponse > -- > > Key: MAPREDUCE-3098 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3098-20110930.txt, MR-3098.1.patch, > MR-3098.2.patch, MR-3098.3.patch, MR-3098.4.patch, MR-3098.5.patch > > > Currently, an application report received by the client from the RM/ASM for a > given application returns the status of the application master. It does not > return the status of the application i.e. whether that particular job > succeeded or failed. > The AM status would be one of FINISHED (SUCCEEDED should be renamed to > FINISHED as AM state does not indicate overall success/failure), FAILED or > KILLED. > The
[jira] [Commented] (MAPREDUCE-2996) Log uberized information into JobHistory and use the same via CompletedJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118362#comment-13118362 ] Hudson commented on MAPREDUCE-2996: --- Integrated in Hadoop-Mapreduce-trunk #846 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/846/]) MAPREDUCE-2996. Add uber-ness information to JobHistory. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177531 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobInitedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/Job20LineHistoryEventEmitter.java > Log uberized information into JobHistory and use the same via CompletedJob > -- > > Key: MAPREDUCE-2996 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2996 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Eagles >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2996.patch, MAPREDUCE-2996.patch, > MAPREDUCE-2996.patch > > > We always print the uberized info on the UI to be false irrespective of > whether it is uberized or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118359#comment-13118359 ] Hudson commented on MAPREDUCE-2791: --- Integrated in Hadoop-Mapreduce-0.23-Build #32 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/32/]) Merge -r 1177486:1177487 from trunk to branch-0.23 to fix MAPREDUCE-2791. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177490 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/capacity-scheduler.xml.template * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRBuilderUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/capacity-scheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/dynamic-scheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-projec
[jira] [Commented] (MAPREDUCE-2996) Log uberized information into JobHistory and use the same via CompletedJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118356#comment-13118356 ] Hudson commented on MAPREDUCE-2996: --- Integrated in Hadoop-Mapreduce-0.23-Build #32 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/32/]) Merge -r 1177530:1177531 from trunk to branch-0.23 to fix MAPREDUCE-2996. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177532 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/capacity-scheduler.xml.template * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobInitedEvent.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/capacity-scheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/dynamic-scheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/fairscheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches
[jira] [Commented] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
ix For: 0.23.0 > > Attachments: MAPREDUCE-3098-20110930.txt, MR-3098.1.patch, > MR-3098.2.patch, MR-3098.3.patch, MR-3098.4.patch, MR-3098.5.patch > > > Currently, an application report received by the client from the RM/ASM for a > given application returns the status of the application master. It does not > return the status of the application i.e. whether that particular job > succeeded or failed. > The AM status would be one of FINISHED (SUCCEEDED should be renamed to > FINISHED as AM state does not indicate overall success/failure), FAILED or > KILLED. > The final state sent by the AM to the RM in the > FinishApplicationMasterRequest should be exposed to the client as > ApplicationState. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118355#comment-13118355 ] Hudson commented on MAPREDUCE-2779: --- Integrated in Hadoop-Mapreduce-0.23-Build #32 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/32/]) MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file. Contributed by Ming Ma. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177783 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/JobSplitWriter.java > JobSplitWriter.java can't handle large job.split file > - > > Key: MAPREDUCE-2779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.20.205.0, 0.22.0, 0.23.0 >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 0.22.0, 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2779-0.22.patch, MAPREDUCE-2779-trunk.patch, > MAPREDUCE-2779-trunk.patch > > > We use cascading MultiInputFormat. MultiInputFormat sometimes generates big > job.split used internally by hadoop, sometimes it can go beyond 2GB. > In JobSplitWriter.java, the function that generates such file uses 32bit > signed integer to compute offset into job.split. > writeNewSplits > ... > int prevCount = out.size(); > ... > int currCount = out.size(); > writeOldSplits > ... > long offset = out.size(); > ... > int currLen = out.size(); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118343#comment-13118343 ] Hudson commented on MAPREDUCE-2779: --- Integrated in Hadoop-Hdfs-trunk-Commit #1071 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1071/]) MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file. Contributed by Ming Ma. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=119 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/JobSplitWriter.java > JobSplitWriter.java can't handle large job.split file > - > > Key: MAPREDUCE-2779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.20.205.0, 0.22.0, 0.23.0 >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 0.22.0, 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2779-0.22.patch, MAPREDUCE-2779-trunk.patch, > MAPREDUCE-2779-trunk.patch > > > We use cascading MultiInputFormat. MultiInputFormat sometimes generates big > job.split used internally by hadoop, sometimes it can go beyond 2GB. > In JobSplitWriter.java, the function that generates such file uses 32bit > signed integer to compute offset into job.split. > writeNewSplits > ... > int prevCount = out.size(); > ... > int currCount = out.size(); > writeOldSplits > ... > long offset = out.size(); > ... > int currLen = out.size(); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3132) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3132: --- Status: Patch Available (was: Open) Example usage and manual testing performed mvn -Dtest=TestMRJobs#testRandomWriter test > Running a set of methods in a Single Test Class > --- > > Key: MAPREDUCE-3132 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3132 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-3132.patch > > > Instead of running every test method in a class, limit to specific testing > methods as describe in the link below. > http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html > Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118338#comment-13118338 ] Hudson commented on MAPREDUCE-2779: --- Integrated in Hadoop-Common-trunk-Commit #993 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/993/]) MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file. Contributed by Ming Ma. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=119 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/split/JobSplitWriter.java > JobSplitWriter.java can't handle large job.split file > - > > Key: MAPREDUCE-2779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.20.205.0, 0.22.0, 0.23.0 >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 0.22.0, 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2779-0.22.patch, MAPREDUCE-2779-trunk.patch, > MAPREDUCE-2779-trunk.patch > > > We use cascading MultiInputFormat. MultiInputFormat sometimes generates big > job.split used internally by hadoop, sometimes it can go beyond 2GB. > In JobSplitWriter.java, the function that generates such file uses 32bit > signed integer to compute offset into job.split. > writeNewSplits > ... > int prevCount = out.size(); > ... > int currCount = out.size(); > writeOldSplits > ... > long offset = out.size(); > ... > int currLen = out.size(); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3132) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3132: --- Attachment: MAPREDUCE-3132.patch > Running a set of methods in a Single Test Class > --- > > Key: MAPREDUCE-3132 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3132 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-3132.patch > > > Instead of running every test method in a class, limit to specific testing > methods as describe in the link below. > http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html > Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2719: --- Status: Patch Available (was: Open) Will create additional javac warnings due to apache-rat-plugin warnings. > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-2179.1.patch, MR-2179.2.patch, MR-2179.3.patch, > MR-2719.4.post3098.patch, MR-2719.5.patch, mr-2719.wip.patch > > > With nextgen hadoop (mrv2), it is simple to write non-MR applications. Write > an AplicationMaster (also corresponding simple client), to submit and run a > shell command application in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3132) Running a set of methods in a Single Test Class
Running a set of methods in a Single Test Class --- Key: MAPREDUCE-3132 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3132 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Instead of running every test method in a class, limit to specific testing methods as describe in the link below. http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3050: --- Status: Open (was: Patch Available) Canceling patch to do some more refactoring. Separating Scheduler*Report from Scheduler*, because, well Scheduler* is not really a report, so it is not as clean as I would like. > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2988: --- Attachment: MR-2988.txt Some java tests are added in. It would be good when adding in mock-container-executor if you could make it executable. The test could should be able to handle it either way, but I would prefer it to be executable by default. > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2988: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt, MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-2719: --- Attachment: MR-2719.5.patch Latest re-based patch. > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-2179.1.patch, MR-2179.2.patch, MR-2179.3.patch, > MR-2719.4.post3098.patch, MR-2719.5.patch, mr-2719.wip.patch > > > With nextgen hadoop (mrv2), it is simple to write non-MR applications. Write > an AplicationMaster (also corresponding simple client), to submit and run a > shell command application in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-2779: --- Resolution: Fixed Fix Version/s: 0.24.0 0.23.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this to 0.22, 0.23, and trunk. Thank you Ming. > JobSplitWriter.java can't handle large job.split file > - > > Key: MAPREDUCE-2779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.20.205.0, 0.22.0, 0.23.0 >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 0.22.0, 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2779-0.22.patch, MAPREDUCE-2779-trunk.patch, > MAPREDUCE-2779-trunk.patch > > > We use cascading MultiInputFormat. MultiInputFormat sometimes generates big > job.split used internally by hadoop, sometimes it can go beyond 2GB. > In JobSplitWriter.java, the function that generates such file uses 32bit > signed integer to compute offset into job.split. > writeNewSplits > ... > int prevCount = out.size(); > ... > int currCount = out.size(); > writeOldSplits > ... > long offset = out.size(); > ... > int currLen = out.size(); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118252#comment-13118252
]
Robert Joseph Evans commented on MAPREDUCE-2988:
@Vinod
The code here is a bit more complex then I expected. None of the tests come
even close to compiling, and as designed none of the tests have a chance of
passing unless I can somehow write to ${HADOOP_CONF_DIR}/taskcontroller.cfg,
which by default is pointing to /. Instead I would like to take a different
route and "mock" out the executable, so that I can get back the data I care
about. Is this acceptable to you?
> Reenable TestLinuxContainerExecutor reflecting the current NM code.
>
>
> Key: MAPREDUCE-2988
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, security, test
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Eric Payne
>Assignee: Robert Joseph Evans
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2988.txt
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> TestLinuxContainerExecutor is currently disabled completely.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3056) Jobs are failing when those are submitted by other users
[ https://issues.apache.org/jira/browse/MAPREDUCE-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118229#comment-13118229 ] Aaron T. Myers commented on MAPREDUCE-3056: --- Hey Devaraj, it should be possible to write a test for this, right? Or am I missing something that would make doing so difficult? > Jobs are failing when those are submitted by other users > > > Key: MAPREDUCE-3056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Devaraj K >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3056.patch > > > MR cluster is started by the user 'root'. If any other users other than > 'root' submit a job, it is failing always. > Find the conatiner logs in the comments section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2980) Fetch failures and other related issues in Jetty 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118184#comment-13118184 ] Todd Lipcon commented on MAPREDUCE-2980: Sent a note to mapreduce-dev@ to solicit more opinions on this issue. We have rolled out this patch to some customers and into QA for our next CDH release. > Fetch failures and other related issues in Jetty 6.1.26 > --- > > Key: MAPREDUCE-2980 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2980 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.205.0, 0.23.0 >Reporter: Todd Lipcon >Priority: Critical > > Since upgrading Jetty from 6.1.14 to 6.1.26 we've had a ton of HTTP-related > issues, including: > - Much higher incidence of fetch failures > - A few strange file-descriptor related bugs (eg MAPREDUCE-2389) > - A few unexplained issues where long "fsck"s on the NameNode drop out > halfway through with a ClosedChannelException > Stress tests with 1Map x 1Reduce sleep jobs reliably reproduce fetch > failures at a rate of about 1 per million on a 25 node test cluster. These > problems are all new since the upgrade from 6.1.14. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118167#comment-13118167 ] Prashant Sharma commented on MAPREDUCE-3131: I am thinking of good and clean(and easy) way of starting of MRV2 Cluster using scripts. I will soon submit code and documentation for review. Till then take a look at http://github.com/prashantiiith/Scripts/wiki/Setup-required-to-play-with-Hadoop-MRV2 thanks prashant > Docs and Scripts for setting up single node MRV2 cluster. > -- > > Key: MAPREDUCE-3131 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Reporter: Prashant Sharma > Fix For: 0.23.0 > > Original Estimate: 168h > Remaining Estimate: 168h > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2600) MR-279: simplify the jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118166#comment-13118166 ] Luke Lu commented on MAPREDUCE-2600: I'd like to get some consensus on this issue. What do people think? This is a fairly large source code reorg that's potentially disruptive. We'd better do it sooner than later or not do it at all. > MR-279: simplify the jars > -- > > Key: MAPREDUCE-2600 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2600 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Owen O'Malley >Assignee: Luke Lu > > Currently the MR-279 mapreduce project generates 59 jars from 59 source > roots, which can be dramatically simplified. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
Docs and Scripts for setting up single node MRV2 cluster. -- Key: MAPREDUCE-3131 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Prashant Sharma -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3050: --- Attachment: MR-3050.txt > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3050: --- Status: Patch Available (was: Open) > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3050: --- Status: Open (was: Patch Available) Canceling patch to upmerge and rename the one class. > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118119#comment-13118119 ] Robert Joseph Evans commented on MAPREDUCE-2988: Will do. Should have a patch shortly. > Reenable TestLinuxContainerExecutor reflecting the current NM code. > > > Key: MAPREDUCE-2988 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security, test >Affects Versions: 0.23.0, 0.24.0 >Reporter: Eric Payne >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2988.txt > > Original Estimate: 72h > Remaining Estimate: 72h > > TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3056) Jobs are failing when those are submitted by other users
[ https://issues.apache.org/jira/browse/MAPREDUCE-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118115#comment-13118115 ] Devaraj K commented on MAPREDUCE-3056: -- I have verified it manually, jobs submitted by other users are executing fine. Please review this. > Jobs are failing when those are submitted by other users > > > Key: MAPREDUCE-3056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Devaraj K >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3056.patch > > > MR cluster is started by the user 'root'. If any other users other than > 'root' submit a job, it is failing always. > Find the conatiner logs in the comments section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3129) JsonFactory can be created only once and used for every next request to create JsonGenerator inside JMXJsonServlet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3129:
-
Status: Open (was: Patch Available)
> JsonFactory can be created only once and used for every next request to
> create JsonGenerator inside JMXJsonServlet
> ---
>
> Key: MAPREDUCE-3129
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3129
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Minor
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3129.patch
>
>
> 1. Currently JMXJsonServlet creates JsonFactory for every http request.
> Its not efficient.
> JsonFactory can be created only once and used for every next request to
> create JsonGenerator.
> 2. Also following null check is not required.
> {code}
> if (mBeanServer == null) {
> jg.writeStringField("result", "ERROR");
> jg.writeStringField("message", "No MBeanServer could be found");
> jg.close();
> LOG.error("No MBeanServer could be found.");
> response.setStatus(HttpServletResponse.SC_NOT_FOUND);
> return;
> }
> {code}
> Because ManagementFactory.getPlatformMBeanServer(); will not return null.
> 3. Move the following code to finally so that any exception should not cause
> skipping of close method on JsonGenerator
> {code}
> jg.close();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3129) JsonFactory can be created only once and used for every next request to create JsonGenerator inside JMXJsonServlet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118098#comment-13118098
]
Robert Joseph Evans commented on MAPREDUCE-3129:
The patch looks good to me. +1 (non-binding).
It looks like test-patch is not applying the change because the code changed is
in common and not under hadoop-mapreduce-project. You may want to close this
JIRA and file a new one in HADOOP with the patch in it.
> JsonFactory can be created only once and used for every next request to
> create JsonGenerator inside JMXJsonServlet
> ---
>
> Key: MAPREDUCE-3129
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3129
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Minor
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3129.patch
>
>
> 1. Currently JMXJsonServlet creates JsonFactory for every http request.
> Its not efficient.
> JsonFactory can be created only once and used for every next request to
> create JsonGenerator.
> 2. Also following null check is not required.
> {code}
> if (mBeanServer == null) {
> jg.writeStringField("result", "ERROR");
> jg.writeStringField("message", "No MBeanServer could be found");
> jg.close();
> LOG.error("No MBeanServer could be found.");
> response.setStatus(HttpServletResponse.SC_NOT_FOUND);
> return;
> }
> {code}
> Because ManagementFactory.getPlatformMBeanServer(); will not return null.
> 3. Move the following code to finally so that any exception should not cause
> skipping of close method on JsonGenerator
> {code}
> jg.close();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2996) Log uberized information into JobHistory and use the same via CompletedJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118076#comment-13118076 ] Jonathan Eagles commented on MAPREDUCE-2996: Thanks, Arun. > Log uberized information into JobHistory and use the same via CompletedJob > -- > > Key: MAPREDUCE-2996 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2996 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Eagles >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2996.patch, MAPREDUCE-2996.patch, > MAPREDUCE-2996.patch > > > We always print the uberized info on the UI to be false irrespective of > whether it is uberized or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118072#comment-13118072 ] Robert Joseph Evans commented on MAPREDUCE-3050: The reason the patch is smaller and simpler is because in the first patch I renamed SchedulerApp to SchedulerAppImpl. SchedulerApp is 20k so when the patch deletes and adds back in the same code it adds 40k to the size of the patch. If you want to change the names I can, it is not a big deal, but I don't think that Scheduler*/Scheduler*Impl really indicate the design pattern being used. I am not trying to separate out interface from implementation. There is just one implementation and no place in the code to allow another one to be inserted through configuration. I am just trying to expose a read only subset of the information about this data structure. After making the change to SchedulerNodeReport, I think it would be best to make the new interface SchedulerAppReport as well. This fits in with the naming conventions used elsewhere. I think I will upload a new patch with it changed to Report. > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3130) Output files are not moved to job output directory after task completion
Output files are not moved to job output directory after task completion
Key: MAPREDUCE-3130
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3130
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 0.20.204.0
Environment: NA
Reporter: Bhallamudi Venkata Siva Kamesh
When MultipleOutputs (OutputFormat set to TextOutputFormat) is used to write
output files to different namedOutputFiles, turned off normal output file
generation(i.e.*part-r-<#reduce>*) by setting OutputFormat as NullOutputFormat,
task output is not moved to job output directory
({color:blue}${mapred.output.dir}{color}).
After the job completion, found output files in
{color:red}${mapred.output.dir}/_temporary/_attempt--<#reduce>{color}
(task output directory) rather than in the
{color:blue}${mapred.output.dir}{color} (job output dir)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3130) Output files are not moved to job output directory after task completion
[
https://issues.apache.org/jira/browse/MAPREDUCE-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118069#comment-13118069
]
Bhallamudi Venkata Siva Kamesh commented on MAPREDUCE-3130:
---
When we set OutputFormat as NullOutputFormal, after the completion of task's
execution, task checks, whether it needs to commit the task output. If commit
is required, all the output files will be moved from *task output dir* to *job
output dir* and after that *_temporary* directory will be deleted. But for
NullOutputFormat, commit is not required, and hence output files remains in the
*_temporary* folder itself.
> Output files are not moved to job output directory after task completion
>
>
> Key: MAPREDUCE-3130
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3130
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
>Affects Versions: 0.20.204.0
> Environment: NA
>Reporter: Bhallamudi Venkata Siva Kamesh
>
> When MultipleOutputs (OutputFormat set to TextOutputFormat) is used to write
> output files to different namedOutputFiles, turned off normal output file
> generation(i.e.*part-r-<#reduce>*) by setting OutputFormat as
> NullOutputFormat, task output is not moved to job output directory
> ({color:blue}${mapred.output.dir}{color}).
>
> After the job completion, found output files in
> {color:red}${mapred.output.dir}/_temporary/_attempt--<#reduce>{color}
> (task output directory) rather than in the
> {color:blue}${mapred.output.dir}{color} (job output dir)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2986) Multiple node managers support for the MiniYARNCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2986: --- Status: Open (was: Patch Available) This patch has bugs which is causing the tests' to hang at integration tests, the first one being TestMROldApiJobs. Please don't submit this patch to Jenkins, one of the builds is stuck and I'm afraid it is going to fill up the disks. Will see how to stop that. Anupam, in the mean while, please run the tests locally with -Dtest.logs=false and try fixing the bug. Thanks! > Multiple node managers support for the MiniYARNCluster > -- > > Key: MAPREDUCE-2986 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2986 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Anupam Seth >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: BZ-4805926-branch-0_23.patch, > MAPREDUCE-2986-branch-0_23.patch, MAPREDUCE-2986-branch-0_23.patch, > MAPREDUCE-2986-branch-0_23.patch, MAPREDUCE-2986-branch-0_23.patch, > MAPREDUCE-2986-branch-0_23.patch, MAPREDUCE-2986-branch-0_23.patch, > MAPREDUCE-2986-branch-0_23.patch > > > The current MiniYARNCluster can only support 1 node manager, which is not > enough for the full test purposes. > Would like to have a simulator that can support multiple node managers as the > real scenario. This might be beneficial for hadoop users, testers and > developers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
doop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/event/RMAppAttemptUnregistrationEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestASMStateMachine.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestApplicationMasterExpiry.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java > Report Application status as well as ApplicationMaster status in > GetApplicationReportResponse > -- > > Key: MAPREDUCE-3098 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3098-20110930.txt, MR-3098.1.patch, > MR-3098.2.patch, MR-3098.3.patch, MR-3098.4.patch, MR-3098.5.patch > > > Currently, an application report received by the client from the RM/ASM for a > given application returns the status of the application master. It does not > return the status of the application i.e. whether that particular job > succeeded or failed. > The AM status would be one of FINISHED (SUCCEEDED should be renamed to > FINISHED as AM state does not indicate overall success/failure), FAILED or > KILLED.
[jira] [Commented] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
doop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/event/RMAppAttemptUnregistrationEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMLaunchFailure.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestASMStateMachine.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestApplicationMasterExpiry.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestSchedulerNegotiator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java > Report Application status as well as ApplicationMaster status in > GetApplicationReportResponse > -- > > Key: MAPREDUCE-3098 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3098-20110930.txt, MR-3098.1.patch, > MR-3098.2.patch, MR-3098.3.patch, MR-3098.4.patch, MR-3098.5.patch > > > Currently, an application report received by the client from the RM/ASM for a > given application returns the status of the application master. It does not > return the status of the application i.e. whether that particular job > succeeded or failed. > The AM status would be one of FINISHED (SUCCEEDED should be renamed to > FINISHED as AM state does not indicate overall success/failure), FAILED or > KILLED.
[jira] [Commented] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118051#comment-13118051 ] Hudson commented on MAPREDUCE-2791: --- Integrated in Hadoop-Hdfs-0.23-Build #25 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/25/]) Merge -r 1177486:1177487 from trunk to branch-0.23 to fix MAPREDUCE-2791. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177490 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/capacity-scheduler.xml.template * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRBuilderUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/capacity-scheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/dynamic-scheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/cont
[jira] [Commented] (MAPREDUCE-2996) Log uberized information into JobHistory and use the same via CompletedJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118050#comment-13118050 ] Hudson commented on MAPREDUCE-2996: --- Integrated in Hadoop-Hdfs-0.23-Build #25 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/25/]) Merge -r 1177530:1177531 from trunk to branch-0.23 to fix MAPREDUCE-2996. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177532 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf/capacity-scheduler.xml.template * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobInitedEvent.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/capacity-scheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/dynamic-scheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/fairscheduler * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.
[jira] [Updated] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
[ https://issues.apache.org/jira/browse/MAPREDUCE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3098: --- Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this to trunk and branch-0.23. Thanks Hitesh! > Report Application status as well as ApplicationMaster status in > GetApplicationReportResponse > -- > > Key: MAPREDUCE-3098 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3098-20110930.txt, MR-3098.1.patch, > MR-3098.2.patch, MR-3098.3.patch, MR-3098.4.patch, MR-3098.5.patch > > > Currently, an application report received by the client from the RM/ASM for a > given application returns the status of the application master. It does not > return the status of the application i.e. whether that particular job > succeeded or failed. > The AM status would be one of FINISHED (SUCCEEDED should be renamed to > FINISHED as AM state does not indicate overall success/failure), FAILED or > KILLED. > The final state sent by the AM to the RM in the > FinishApplicationMasterRequest should be exposed to the client as > ApplicationState. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2996) Log uberized information into JobHistory and use the same via CompletedJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118027#comment-13118027 ] Hudson commented on MAPREDUCE-2996: --- Integrated in Hadoop-Hdfs-trunk #816 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/816/]) MAPREDUCE-2996. Add uber-ness information to JobHistory. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177531 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro/Events.avpr * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobInitedEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/Job20LineHistoryEventEmitter.java > Log uberized information into JobHistory and use the same via CompletedJob > -- > > Key: MAPREDUCE-2996 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2996 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Eagles >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2996.patch, MAPREDUCE-2996.patch, > MAPREDUCE-2996.patch > > > We always print the uberized info on the UI to be false irrespective of > whether it is uberized or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118031#comment-13118031 ] Hudson commented on MAPREDUCE-2791: --- Integrated in Hadoop-Hdfs-trunk #816 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/816/]) MAPREDUCE-2791. Added missing info on 'job -status' output. Contributed by Devaraj K. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177487 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/JobReport.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/records/impl/pb/JobReportPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRBuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java > [MR-279] Missing/incorrect info on job -status CLI > --- > > Key: MAPREDUCE-2791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2791-1.patch, MAPREDUCE-2791-1.patch, > MAPREDUCE-2791-2.patch, MAPREDUCE-2791-3.patch, MAPREDUCE-2791-4.patch, > MAPREDUCE-2791-5.patch, MAPREDUCE-2791.patch, MAPREDUCE-2791.patch, mvn > install.log > > > There are a couple of details missing/incorrect on the job -status command > line output for completed jobs: > 1. Incorrect job file > 2. map() completion is always 0 > 3. reduce() completion is always set to 0 > 4. history URL is empty > 5. Missing launched map tasks > 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3013) Remove YarnConfiguration.YARN_SECURITY_INFO
[ https://issues.apache.org/jira/browse/MAPREDUCE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118024#comment-13118024 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3013: Alright, things are back to normality. Compiled and tested on a single node secure setup. It works. > Remove YarnConfiguration.YARN_SECURITY_INFO > --- > > Key: MAPREDUCE-3013 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3013 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3013-20110930.txt > > > We don't need this anymore since RPC client uses SecurityUtil to pick it up > via going through the providers for SecurityInfo interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3129) JsonFactory can be created only once and used for every next request to create JsonGenerator inside JMXJsonServlet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118021#comment-13118021
]
Hadoop QA commented on MAPREDUCE-3129:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12497151/MAPREDUCE-3129.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/909//console
This message is automatically generated.
> JsonFactory can be created only once and used for every next request to
> create JsonGenerator inside JMXJsonServlet
> ---
>
> Key: MAPREDUCE-3129
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3129
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Minor
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3129.patch
>
>
> 1. Currently JMXJsonServlet creates JsonFactory for every http request.
> Its not efficient.
> JsonFactory can be created only once and used for every next request to
> create JsonGenerator.
> 2. Also following null check is not required.
> {code}
> if (mBeanServer == null) {
> jg.writeStringField("result", "ERROR");
> jg.writeStringField("message", "No MBeanServer could be found");
> jg.close();
> LOG.error("No MBeanServer could be found.");
> response.setStatus(HttpServletResponse.SC_NOT_FOUND);
> return;
> }
> {code}
> Because ManagementFactory.getPlatformMBeanServer(); will not return null.
> 3. Move the following code to finally so that any exception should not cause
> skipping of close method on JsonGenerator
> {code}
> jg.close();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3128) WebAppContext should also be stopped and cleared
[
https://issues.apache.org/jira/browse/MAPREDUCE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118020#comment-13118020
]
Hadoop QA commented on MAPREDUCE-3128:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12497150/MAPREDUCE-3128.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/910//console
This message is automatically generated.
> WebAppContext should also be stopped and cleared
>
>
> Key: MAPREDUCE-3128
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3128
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3128.patch
>
>
> 1. If listener stop method throws any exception then the webserver stop
> method will not be called
> {code}
> public void stop() throws Exception {
> listener.close();
> webServer.stop();
> }
> {code}
> 2. also, WebAppContext stores all the context attributes, which does not get
> cleared if only webServer is stopped.
> so following calls are necessary to ensure clean and complete stop.
> {code}
> webAppContext.clearAttributes();
> webAppContext.stop();
> {code}
> 3. Also the WebAppContext display name can be the name passed to HttpServer
> instance.
> {code}
> webAppContext.setDisplayName(name);
> {code}
> instead of
> {code}
> webAppContext.setDisplayName("WepAppsContext");
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
