[jira] [Updated] (MAPREDUCE-4195) With invalid queueName request param, jobqueue_details.jsp shows NPE
[ https://issues.apache.org/jira/browse/MAPREDUCE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-4195: - Attachment: MR-4195 Hi Harsh, Yes, I am aware that this is not a documented way but this obviously has happened given MAPREDUCE-3674. Grant license to ASF is selected. Thanks, Gera With invalid queueName request param, jobqueue_details.jsp shows NPE Key: MAPREDUCE-4195 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4195 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 1.0.0 Reporter: Gera Shegalov Priority: Critical Fix For: 1.1.0 Attachments: MR-4195, MR-4195 When you access /jobqueue_details.jsp manually, instead of via a link, it has queueName set to null internally and this goes for a lookup into the scheduling info maps as well. As a result, if using FairScheduler, a Pool with String name = null gets created and this brings the scheduler down. I have not tested what happens to the CapacityScheduler, but ideally if no queueName is set in that jsp, it should fall back to 'default'. Otherwise, this brings down the JobTracker completely. FairScheduler must also add a check to not create a pool with 'null' name. The following is the strace that ensues: {code} ERROR org.mortbay.log: /jobqueue_details.jsp java.lang.NullPointerException at org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call heartbeat from XYZ:MNOP: error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95) at org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68) at java.util.Arrays.mergeSort(Unknown Source) at java.util.Arrays.sort(Unknown Source) at java.util.Collections.sort(Unknown Source) at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435) at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on /bin/bash
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262437#comment-13262437 ] Radim Kolar commented on MAPREDUCE-4101: There is already configurable shell feature - MRJobConfig.MAPRED_ADMIN_USER_SHELL it is just not used for anything except putting into environment in TaskAttemptImpl. You want to use this setting for every container launch? nodemanager depends on /bin/bash Key: MAPREDUCE-4101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.1, trunk Environment: FreeBSD 8.2 / 64 bit Reporter: Radim Kolar Labels: patch Attachments: bash-replace-by-sh.txt Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough. i found 2 cases: 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin) 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh] this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3873) Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xieguiming updated MAPREDUCE-3873: -- Attachment: MAPREDUCE-3873.patch Nodemanager is not getting decommisioned if the absolute ip is given in exclude file. - Key: MAPREDUCE-3873 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3873 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Nishan Shetty Priority: Minor Attachments: MAPREDUCE-3873.patch Configure absolute ip in yarn.resourcemanager.nodes.exclude-path and try to decommission the node. It is not getting decommisioned.But if the hostname is given, decommissioning is happening. I have also given the ip-host mapping of each machine in /etc/hosts.(i,e in every machine the other machines ip-host mapping is specified). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3873) Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xieguiming updated MAPREDUCE-3873: -- Status: Patch Available (was: Open) The patch will support that we config either hostname or ip in the yarn.resourcemanager.nodes.exclude-path and yarn.resourcemanager.nodes.include-path file. Nodemanager is not getting decommisioned if the absolute ip is given in exclude file. - Key: MAPREDUCE-3873 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3873 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Nishan Shetty Priority: Minor Attachments: MAPREDUCE-3873.patch Configure absolute ip in yarn.resourcemanager.nodes.exclude-path and try to decommission the node. It is not getting decommisioned.But if the hostname is given, decommissioning is happening. I have also given the ip-host mapping of each machine in /etc/hosts.(i,e in every machine the other machines ip-host mapping is specified). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262571#comment-13262571 ] Hudson commented on MAPREDUCE-4194: --- Integrated in Hadoop-Hdfs-0.23-Build #239 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/239/]) svn merge -c 1330552 FIXES: MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330556) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330556 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java ConcurrentModificationError in DirectoryCollection -- Key: MAPREDUCE-4194 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch As found as part of work on MAPREDUCE-4169, it is possible for a ConcurrentModificationException to be thrown upon disk failure. DirectoryCollection hands out its internal list structure that is accessed across multiple threads. Upon disk failure its internal list is modified, invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262570#comment-13262570 ] Hudson commented on MAPREDUCE-3613: --- Integrated in Hadoop-Hdfs-0.23-Build #239 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/239/]) merge -r 1330560:1330561 from branch-2. FIXES: MAPREDUCE-3613 (Revision 1330562) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330562 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java web service calls header contains 2 content types - Key: MAPREDUCE-3613 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-3613.patch when doing requesting info from the web services rest API, curl seems to return content-type of both text and json or xml: Accept: application/xml HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4199) Not easy to search the property because of splited into many parts.
xieguiming created MAPREDUCE-4199: - Summary: Not easy to search the property because of splited into many parts. Key: MAPREDUCE-4199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4199 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.1 Environment: suse Reporter: xieguiming Priority: Minor The property is splited into many parts. such as: public static final String RM_NODES_EXCLUDE_FILE_PATH = RM_PREFIX + nodes.exclude-path; It is inconvenient for code reader to search the code by the property name although there are some common parts for reuse. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262606#comment-13262606 ] Hudson commented on MAPREDUCE-4193: --- Integrated in Hadoop-Hdfs-trunk #1026 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/]) MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml. Contributed by Patrick Hunt. (Revision 1330427) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330427 Files : * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl/configuration.xsl * /hadoop/common/trunk/hadoop-project/src/site/site.xml broken doc link for yarn-default.xml in site.xml Key: MAPREDUCE-4193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Reporter: Patrick Hunt Assignee: Patrick Hunt Fix For: 2.0.0 Attachments: MAPREDUCE-4193.patch the link to yarn-default.xml in site.xml is incorrect, generated docs link is broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262605#comment-13262605 ] Hudson commented on MAPREDUCE-3613: --- Integrated in Hadoop-Hdfs-trunk #1026 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/]) MAPREDUCE-3613. web service calls header contains 2 content types (tgraves) (Revision 1330560) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330560 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java web service calls header contains 2 content types - Key: MAPREDUCE-3613 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-3613.patch when doing requesting info from the web services rest API, curl seems to return content-type of both text and json or xml: Accept: application/xml HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262607#comment-13262607 ] Hudson commented on MAPREDUCE-4194: --- Integrated in Hadoop-Hdfs-trunk #1026 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/]) MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330552 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java ConcurrentModificationError in DirectoryCollection -- Key: MAPREDUCE-4194 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch As found as part of work on MAPREDUCE-4169, it is possible for a ConcurrentModificationException to be thrown upon disk failure. DirectoryCollection hands out its internal list structure that is accessed across multiple threads. Upon disk failure its internal list is modified, invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262619#comment-13262619 ] Hudson commented on MAPREDUCE-3613: --- Integrated in Hadoop-Mapreduce-trunk #1061 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/]) MAPREDUCE-3613. web service calls header contains 2 content types (tgraves) (Revision 1330560) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330560 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java web service calls header contains 2 content types - Key: MAPREDUCE-3613 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-3613.patch when doing requesting info from the web services rest API, curl seems to return content-type of both text and json or xml: Accept: application/xml HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262620#comment-13262620 ] Hudson commented on MAPREDUCE-4193: --- Integrated in Hadoop-Mapreduce-trunk #1061 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/]) MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml. Contributed by Patrick Hunt. (Revision 1330427) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330427 Files : * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl/configuration.xsl * /hadoop/common/trunk/hadoop-project/src/site/site.xml broken doc link for yarn-default.xml in site.xml Key: MAPREDUCE-4193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Reporter: Patrick Hunt Assignee: Patrick Hunt Fix For: 2.0.0 Attachments: MAPREDUCE-4193.patch the link to yarn-default.xml in site.xml is incorrect, generated docs link is broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262621#comment-13262621 ] Hudson commented on MAPREDUCE-4194: --- Integrated in Hadoop-Mapreduce-trunk #1061 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/]) MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330552 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java ConcurrentModificationError in DirectoryCollection -- Key: MAPREDUCE-4194 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch As found as part of work on MAPREDUCE-4169, it is possible for a ConcurrentModificationException to be thrown upon disk failure. DirectoryCollection hands out its internal list structure that is accessed across multiple threads. Upon disk failure its internal list is modified, invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4191) capacity scheduler: job unexpectedly exceeds queue capacity limit by one task
[ https://issues.apache.org/jira/browse/MAPREDUCE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned MAPREDUCE-4191: Assignee: Thomas Graves capacity scheduler: job unexpectedly exceeds queue capacity limit by one task - Key: MAPREDUCE-4191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4191 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Thomas Graves While testing the queue capacity limits, it appears that the job can exceed the queue capacity limit by one task while the user limit factor is 1. It's not clear to me why this is. Here is the steps to reproduce: 1) set yarn.app.mapreduce.am.resource.mb to 2048 (default value) 2) set yarn.scheduler.capacity.root.default.user-limit-factor to 1.0 (default) 3) set yarn.scheduler.capacity.root.default.capacity to 90 (%) 4) For a cluster with capacity of 56G, 90% rounded up is 51. 5) submit a job with large number of tasks, each task using 1G memory. 6) webui shows that the used resource is 52 G, which is 92.9% of the cluster capacity (instead of the expected 90%), and 103.2% of the queue capacity (instead of the expected 100%). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3873) Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262623#comment-13262623 ] Hadoop QA commented on MAPREDUCE-3873: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524425/MAPREDUCE-3873.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2314//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2314//console This message is automatically generated. Nodemanager is not getting decommisioned if the absolute ip is given in exclude file. - Key: MAPREDUCE-3873 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3873 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Nishan Shetty Priority: Minor Attachments: MAPREDUCE-3873.patch Configure absolute ip in yarn.resourcemanager.nodes.exclude-path and try to decommission the node. It is not getting decommisioned.But if the hostname is given, decommissioning is happening. I have also given the ip-host mapping of each machine in /etc/hosts.(i,e in every machine the other machines ip-host mapping is specified). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4155) Provide support for these yarn commands
[ https://issues.apache.org/jira/browse/MAPREDUCE-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-4155: - Attachment: MAPREDUCE-4155.patch Provide support for these yarn commands --- Key: MAPREDUCE-4155 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4155 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0 Reporter: Devaraj K Assignee: Devaraj K Attachments: MAPREDUCE-4155.patch 1. status app-id 2. kill app-id (Already issue present with Id : MAPREDUCE-3793) 3. list-apps [all] 4. nodes-report -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4155) Provide support for these yarn commands
[ https://issues.apache.org/jira/browse/MAPREDUCE-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-4155: - Affects Version/s: 3.0.0 Status: Patch Available (was: Open) I have attached patch for this. Please review and give your comments on this. Provide support for these yarn commands --- Key: MAPREDUCE-4155 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4155 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Attachments: MAPREDUCE-4155.patch 1. status app-id 2. kill app-id (Already issue present with Id : MAPREDUCE-3793) 3. list-apps [all] 4. nodes-report -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4200) packaging tar ball of trunk failed
Tomohiko Kinebuchi created MAPREDUCE-4200: - Summary: packaging tar ball of trunk failed Key: MAPREDUCE-4200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: trunk Environment: Mac OS X Lion 10.7.3 Reporter: Tomohiko Kinebuchi A command mvn clean package -Dtar -DskipTests executed on the root directory hadoop-common failed. Its output logs are in an attached file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4200) packaging tar ball of trunk failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomohiko Kinebuchi updated MAPREDUCE-4200: -- Attachment: MAPREDUCE-4200.error.log packaging tar ball of trunk failed -- Key: MAPREDUCE-4200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: trunk Environment: Mac OS X Lion 10.7.3 Reporter: Tomohiko Kinebuchi Attachments: MAPREDUCE-4200.error.log A command mvn clean package -Dtar -DskipTests executed on the root directory hadoop-common failed. Its output logs are in an attached file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4155) Provide support for these yarn commands
[ https://issues.apache.org/jira/browse/MAPREDUCE-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262667#comment-13262667 ] Hadoop QA commented on MAPREDUCE-4155: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524439/MAPREDUCE-4155.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2315//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2315//console This message is automatically generated. Provide support for these yarn commands --- Key: MAPREDUCE-4155 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4155 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Attachments: MAPREDUCE-4155.patch 1. status app-id 2. kill app-id (Already issue present with Id : MAPREDUCE-3793) 3. list-apps [all] 4. nodes-report -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4200) packaging tar ball of trunk failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262688#comment-13262688 ] Radim Kolar commented on MAPREDUCE-4200: did you looked at builds.apache.org if your error is Mac OS specific? packaging tar ball of trunk failed -- Key: MAPREDUCE-4200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: trunk Environment: Mac OS X Lion 10.7.3 Reporter: Tomohiko Kinebuchi Attachments: MAPREDUCE-4200.error.log A command mvn clean package -Dtar -DskipTests executed on the root directory hadoop-common failed. Its output logs are in an attached file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4200) packaging tar ball of trunk failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262693#comment-13262693 ] Tomohiko Kinebuchi commented on MAPREDUCE-4200: --- @Radim Thanks for your comment. Results of the last five builds are all failed. https://builds.apache.org/job/Hadoop-Mapreduce-trunk/ So this issue is not Mac OS specific. packaging tar ball of trunk failed -- Key: MAPREDUCE-4200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: trunk Environment: Mac OS X Lion 10.7.3 Reporter: Tomohiko Kinebuchi Attachments: MAPREDUCE-4200.error.log A command mvn clean package -Dtar -DskipTests executed on the root directory hadoop-common failed. Its output logs are in an attached file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4200) packaging tar ball of trunk failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomohiko Kinebuchi updated MAPREDUCE-4200: -- Description: A command mvn clean package -Dtar -DskipTests executed on the root directory hadoop-common failed. Its output logs are in an attached file. A command mvn clean package -Pdist -DskipTests succeeded. was: A command mvn clean package -Dtar -DskipTests executed on the root directory hadoop-common failed. Its output logs are in an attached file. packaging tar ball of trunk failed -- Key: MAPREDUCE-4200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: trunk Environment: Mac OS X Lion 10.7.3 Reporter: Tomohiko Kinebuchi Attachments: MAPREDUCE-4200.error.log A command mvn clean package -Dtar -DskipTests executed on the root directory hadoop-common failed. Its output logs are in an attached file. A command mvn clean package -Pdist -DskipTests succeeded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4200) packaging tar ball of trunk failed
[ https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262751#comment-13262751 ] Tomohiko Kinebuchi commented on MAPREDUCE-4200: --- From output messages, this ant target executed and failed. https://github.com/apache/hadoop-common/blob/235bcba59b02f8a3234d042010cf74f8dcf4a2fa/hadoop-mapreduce-project/pom.xml#L178 packaging tar ball of trunk failed -- Key: MAPREDUCE-4200 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: trunk Environment: Mac OS X Lion 10.7.3 Reporter: Tomohiko Kinebuchi Attachments: MAPREDUCE-4200.error.log A command mvn clean package -Dtar -DskipTests executed on the root directory hadoop-common failed. Its output logs are in an attached file. A command mvn clean package -Pdist -DskipTests succeeded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4191) capacity scheduler: job unexpectedly exceeds queue capacity limit by one task
[ https://issues.apache.org/jira/browse/MAPREDUCE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262799#comment-13262799 ] Thomas Graves commented on MAPREDUCE-4191: -- I'm still following this through to fully understand, but there is a comment in the code in LeafQueue that tries to explain this: // Note: We aren't considering the current request since there is a fixed // overhead of the AM, but it's a check, not a = check, so... Which I don't totally follow, I guess if you have one job in the queue that is taking the entire capacity, it allows the job to be more like it was in mrv1 and tries not to penalize you for the AM overhead. The AM however is doing the setup and clean tasks where as in mrv1 it would need to allocate a slot for those. The AM may have fixed overhead but that overhead is configurable. I could create an AM with 24G of memory or use the default of 1.5G. Or on the flip side, I have an AM that uses 1.5G, but have a map task that now gets scheduled and uses 24G which puts it way over its capacity. That could affect the queue current usage greatly and seems to break the capacity guarantee. In the case where you say have 2 jobs in the queue, you have 2 app masters, one of which is counted' against your queue and then the other one is not. I do see it beneficial to queues with very small capacities though, as without this they could be stuck without enough resources to run a task. Arun or anyone else familiar with capacity scheduler, if you could provide explanation that would be great. capacity scheduler: job unexpectedly exceeds queue capacity limit by one task - Key: MAPREDUCE-4191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4191 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Thomas Graves While testing the queue capacity limits, it appears that the job can exceed the queue capacity limit by one task while the user limit factor is 1. It's not clear to me why this is. Here is the steps to reproduce: 1) set yarn.app.mapreduce.am.resource.mb to 2048 (default value) 2) set yarn.scheduler.capacity.root.default.user-limit-factor to 1.0 (default) 3) set yarn.scheduler.capacity.root.default.capacity to 90 (%) 4) For a cluster with capacity of 56G, 90% rounded up is 51. 5) submit a job with large number of tasks, each task using 1G memory. 6) webui shows that the used resource is 52 G, which is 92.9% of the cluster capacity (instead of the expected 90%), and 103.2% of the queue capacity (instead of the expected 100%). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4066) To get yarn.app.mapreduce.am.staging-dir value, should set the default value
[ https://issues.apache.org/jira/browse/MAPREDUCE-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-4066: -- Assignee: xieguiming To get yarn.app.mapreduce.am.staging-dir value, should set the default value -- Key: MAPREDUCE-4066 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4066 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, mrv2 Affects Versions: 0.23.1 Environment: client is windows eclipse, server is suse Reporter: xieguiming Assignee: xieguiming Priority: Minor Fix For: 2.0.0 Attachments: MAPREDUCE-4066.patch, MAPREDUCE-4066.patch when submit the job use the windows eclipse, and the yarn.app.mapreduce.am.staging-dir value is null. {code:title=MRApps.java|borderStyle=solid} public static Path getStagingAreaDir(Configuration conf, String user) { return new Path( conf.get(MRJobConfig.MR_AM_STAGING_DIR) + Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT); } {code} should modify to: {code:title=MRApps.java|borderStyle=solid} public static Path getStagingAreaDir(Configuration conf, String user) { return new Path( conf.get(MRJobConfig.MR_AM_STAGING_DIR,/tmp/hadoop-yarn/staging) + Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT); } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4074) Client continuously retries to RM When RM goes down before launching Application Master
[ https://issues.apache.org/jira/browse/MAPREDUCE-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-4074: -- Assignee: xieguiming Client continuously retries to RM When RM goes down before launching Application Master --- Key: MAPREDUCE-4074 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4074 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1 Reporter: Devaraj K Assignee: xieguiming Fix For: 0.23.3 Attachments: MAPREDUCE-4074-1.patch, MAPREDUCE-4074-2.patch, MAPREDUCE-4074-3.patch, MAPREDUCE-4074.patch Client continuously tries to RM and logs the below messages when the RM goes down before launching App Master. I feel exception should be thrown or break the loop after finite no of retries. {code:xml} 28/03/12 07:15:03 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 0 time(s). 28/03/12 07:15:04 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 1 time(s). 28/03/12 07:15:05 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 2 time(s). 28/03/12 07:15:06 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 3 time(s). 28/03/12 07:15:07 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 4 time(s). 28/03/12 07:15:08 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 5 time(s). 28/03/12 07:15:09 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 6 time(s). 28/03/12 07:15:10 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 7 time(s). 28/03/12 07:15:11 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 8 time(s). 28/03/12 07:15:12 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 9 time(s). 28/03/12 07:15:13 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 0 time(s). 28/03/12 07:15:14 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 1 time(s). 28/03/12 07:15:15 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 2 time(s). 28/03/12 07:15:16 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 3 time(s). 28/03/12 07:15:17 INFO ipc.Client: Retrying connect to server: linux-f330.site/10.18.40.182:8032. Already tried 4 time(s). {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3173) MRV2 UI doesn't work properly without internet
[ https://issues.apache.org/jira/browse/MAPREDUCE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262819#comment-13262819 ] Robert Joseph Evans commented on MAPREDUCE-3173: I tried to to set this up, but I am getting some 404 errors for images from the base theme. I don't know if they are critical or not, but I have to assume that they are, and that other themes are missing images too. Could you provide a tgz of what you have under hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery/themes-1.8.16/ instead of the patch as that way we can get all of the images too. MRV2 UI doesn't work properly without internet -- Key: MAPREDUCE-3173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3173 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Attachments: MAPREDUCE-3173.patch, jquery-ui.min-1.8.16.js.gz, jquery.min-1.6.4.js.gz When we try access the MRV2 UI, it is always giving the below message in the UI even if the java script enabled in the browser. {code:xml} This page works best with javascript enabled. {code} It is trying to download these below css/js files from internet and finally ending up with the above message. For loading the page also it is taking long time. {code:title=JQueryUI.java|borderStyle=solid} html. link(join(https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/themes/;, getTheme(), /jquery-ui.css)). link(/static/dt-1.7.5/css/jui-dt.css). script(https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js;). script(https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/jquery-ui.min.js;). {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3173) MRV2 UI doesn't work properly without internet
[ https://issues.apache.org/jira/browse/MAPREDUCE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3173: --- Status: Open (was: Patch Available) MRV2 UI doesn't work properly without internet -- Key: MAPREDUCE-3173 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3173 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Attachments: MAPREDUCE-3173.patch, jquery-ui.min-1.8.16.js.gz, jquery.min-1.6.4.js.gz When we try access the MRV2 UI, it is always giving the below message in the UI even if the java script enabled in the browser. {code:xml} This page works best with javascript enabled. {code} It is trying to download these below css/js files from internet and finally ending up with the above message. For loading the page also it is taking long time. {code:title=JQueryUI.java|borderStyle=solid} html. link(join(https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/themes/;, getTheme(), /jquery-ui.css)). link(/static/dt-1.7.5/css/jui-dt.css). script(https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js;). script(https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/jquery-ui.min.js;). {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4169: --- Resolution: Fixed Fix Version/s: 3.0.0 2.0.0 0.23.3 Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Resolved (was: Patch Available) Thanks John, I put this into trunk, branch-2, and branch-0.23 Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4138) Reduce memory usage of counters due to non-static nested classes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-4138: - Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks for reviewing, Alejandro and Robert. Reduce memory usage of counters due to non-static nested classes Key: MAPREDUCE-4138 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4138 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Tom White Assignee: Tom White Fix For: 2.0.0 Attachments: MAPREDUCE-4138.patch, MAPREDUCE-4138.patch FrameworkCounter is a non-static nested class of FrameworkCounterGroup which means it retains a reference to the outer class, which isn't really needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262894#comment-13262894 ] Jonathan Eagles commented on MAPREDUCE-4169: Awesome, thanks so much for the thorough reviews Bobby and Harsh. Jon Container Logs appear in unsorted order --- Key: MAPREDUCE-4169 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, MAPREDUCE-4169.patch container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear in unsorted order where the order displayed is based on what file was created first. This jira will have the results be displayed in a consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4189: --- Resolution: Fixed Fix Version/s: 3.0.0 2.0.0 0.23.3 Status: Resolved (was: Patch Available) Thanks Devaraj, I have put this into trunk, branch-2, and branch-0.23. TestContainerManagerSecurity is failing --- Key: MAPREDUCE-4189 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: trunk Reporter: Devaraj K Assignee: Devaraj K Priority: Critical Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: MAPREDUCE-4189.patch, MAPREDUCE-4189.patch {code:xml} --- T E S T S --- Running org.apache.hadoop.yarn.server.TestDiskFailures Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec FAILURE! Results : Tests in error: testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity) testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity) testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity) Tests run: 5, Failures: 0, Errors: 3, Skipped: 0 {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4201) Getting PID not working on Windows. Termination of Task/TaskJVM's not working
Bikas Saha created MAPREDUCE-4201: - Summary: Getting PID not working on Windows. Termination of Task/TaskJVM's not working Key: MAPREDUCE-4201 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4201 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha Child Task not reporting PID because of Linux specific shell script implementation. Signaling task termination currently disabled by the initial Windows patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4155) Provide support for these yarn commands
[ https://issues.apache.org/jira/browse/MAPREDUCE-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263114#comment-13263114 ] Ahmed Radwan commented on MAPREDUCE-4155: - Thanks Devaraj for this addition! +1, can you please add docs for these new commands (e.g. in YarnCommands.apt.vm). Provide support for these yarn commands --- Key: MAPREDUCE-4155 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4155 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 2.0.0, 3.0.0 Reporter: Devaraj K Assignee: Devaraj K Attachments: MAPREDUCE-4155.patch 1. status app-id 2. kill app-id (Already issue present with Id : MAPREDUCE-3793) 3. list-apps [all] 4. nodes-report -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4201) Getting PID not working on Windows. Termination of Task/TaskJVM's not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-4201: -- Attachment: MAPREDUCE-4201.patch patch addresses the issues. Using windows built in tasklist and taskkill commands for the process management in util.ProcessTree. Plan to write equivalent of ProcFsBasedProcessTree later on and add equivalent tests. Getting PID not working on Windows. Termination of Task/TaskJVM's not working - Key: MAPREDUCE-4201 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4201 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha Attachments: MAPREDUCE-4201.patch Child Task not reporting PID because of Linux specific shell script implementation. Signaling task termination currently disabled by the initial Windows patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4202) TestYarnClientProtocolProvider is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-4202: --- Attachment: MAPREDUCE-4202.patch Use same type of conditional as yarn's protocol provider. The test and related files haven't been touched in a long time, so I have no idea how this test ever passed unless the CLASSPATH changed such that the service provider started finding the local provider class. TestYarnClientProtocolProvider is broken Key: MAPREDUCE-4202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4202 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.24.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: MAPREDUCE-4202.patch The test fails because a cluster is unexpectedly created with an empty conf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3987) Reduce RM app memory footprint once app has completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned MAPREDUCE-3987: -- Assignee: (was: Jonathan Eagles) Reduce RM app memory footprint once app has completed - Key: MAPREDUCE-3987 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3987 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, resourcemanager Affects Versions: 0.23.1 Reporter: Jason Lowe The ResourceManager holds onto a configurable number of completed applications (yarn.resource.max-completed-applications, defaults to 1), and the memory footprint of these completed applications can be significant. For example, the {{submissionContext}} in RMAppImpl contains references to protocolbuffer objects and other items that probably aren't necessary to keep around once the application has completed. We could significantly reduce the memory footprint of the RM by releasing objects that are no longer necessary once an application completes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4002) MultiFileWordCount job fails if the input path is not from default file system
[ https://issues.apache.org/jira/browse/MAPREDUCE-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263155#comment-13263155 ] Ahmed Radwan commented on MAPREDUCE-4002: - +1, change lgtm. Thanks! MultiFileWordCount job fails if the input path is not from default file system -- Key: MAPREDUCE-4002 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4002 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 0.24.0, 0.23.3 Reporter: Bhallamudi Venkata Siva Kamesh Attachments: MAPREDUCE-4002.patch In the MultiFileWordCount#CombineFileLineRecordReader, filesystem object has been initialized in the following way {noformat}fs = FileSystem.get(context.getConfiguration());{noformat} This causes, *fs* to be initialized with default filesystem. Therefore *fs* searchs for the input files on the default file system, which fails if the input path is from different source. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3871) Allow symlinking in LocalJobRunner DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3871: - Attachment: MAPREDUCE-3871.patch Small change to add a negative test for files that should not be symlinked (check that no symlink is present). Allow symlinking in LocalJobRunner DistributedCache --- Key: MAPREDUCE-3871 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3871 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-3871.patch, MAPREDUCE-3871.patch Currently the LocalJobRunner doesn't create symlinks for files in the DistributedCache. It is safe to create symlinks if files of the same name don't exist. LocalJobRunner should also delete the symlinks when the job has completed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4088) Task stuck in JobLocalizer prevented other tasks on the same node from committing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263197#comment-13263197 ] Ravi Prakash commented on MAPREDUCE-4088: - Courtesy Koji {quote} Each TaskTracker has a single thread for taskCleanup taking work from tasksToCleanup queue. For each task to cleanup, it firsts call checkJobStatusAndWait(action); And inside {noformat} private void checkJobStatusAndWait(TaskTrackerAction action) ... synchronized (runningJobs) { rjob = runningJobs.get(jobId); } if (rjob != null) { synchronized (rjob) { while (rjob.localizing) { rjob.wait(); } } } {noformat} So this thread would wait while the task is being localized. Even if one task is hung on localization, entire cleanup is stopped. {quote} East or west! Koji is the best! Soda lemon ginger pop! Koji is on the top! Yyaayyy yaa yaa for Koji! Task stuck in JobLocalizer prevented other tasks on the same node from committing - Key: MAPREDUCE-4088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4088 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.20.205.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Critical We saw that as a result of HADOOP-6963, one task was stuck in this Thread 23668: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=3, line=446 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) TONS MORE OF THIS SAME LINE - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) . . - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Interpreted frame) ne=451 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects(org.apache.hadoop.conf.Configuration, java.net.URI[], org.apache.hadoop.fs.Path[], long[], boolean[], boolean) @bci=150, line=324 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCache(org.apache.hadoop.conf.Configuration) @bci=40, line=349 (Interpreted frame) 51, line=383 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.runSetup(java.lang.String, java.lang.String, org.apache.hadoop.fs.Path, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=46, line=477 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=20, line=534 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=1, line=531 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=14, line=1082 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.main(java.lang.String[]) @bci=266, line=530 (Interpreted frame) While all other tasks on the same node were stuck in Thread 32141: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) - org.apache.hadoop.mapred.Task.commit(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter, org.apache.hadoop.mapreduce.OutputCommitter) @bci=24, line=980 (Compiled frame) - org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter) @bci=146, line=871 (Interpreted frame) - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=470, line=423 (Interpreted frame) - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=255 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) -
[jira] [Created] (MAPREDUCE-4203) Create equivalent of ProcfsBasedProcessTree for Windows
Bikas Saha created MAPREDUCE-4203: - Summary: Create equivalent of ProcfsBasedProcessTree for Windows Key: MAPREDUCE-4203 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Bikas Saha Assignee: Bikas Saha ProcfsBasedProcessTree is used by the TaskTracker to get process information like memory and cpu usage. This information is used to manage resources etc. The current implementation is based on Linux procfs functionality and hence does not work on other platforms, specifically windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4204) Refactor ProcfsBasedProcessTree to make the resource collection object pluggable
Bikas Saha created MAPREDUCE-4204: - Summary: Refactor ProcfsBasedProcessTree to make the resource collection object pluggable Key: MAPREDUCE-4204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4204 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Bikas Saha Assignee: Bikas Saha Making it a pluggable interface will allow replacing the procfs based implementation with ones for other platforms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4202) TestYarnClientProtocolProvider is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263214#comment-13263214 ] Hadoop QA commented on MAPREDUCE-4202: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524770/MAPREDUCE-4202.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.conf.TestNoDefaultsJobConf +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2316//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2316//console This message is automatically generated. TestYarnClientProtocolProvider is broken Key: MAPREDUCE-4202 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4202 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.24.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: MAPREDUCE-4202.patch The test fails because a cluster is unexpectedly created with an empty conf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3871) Allow symlinking in LocalJobRunner DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263217#comment-13263217 ] Hadoop QA commented on MAPREDUCE-3871: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524772/MAPREDUCE-3871.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2317//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2317//console This message is automatically generated. Allow symlinking in LocalJobRunner DistributedCache --- Key: MAPREDUCE-3871 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3871 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distributed-cache Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-3871.patch, MAPREDUCE-3871.patch Currently the LocalJobRunner doesn't create symlinks for files in the DistributedCache. It is safe to create symlinks if files of the same name don't exist. LocalJobRunner should also delete the symlinks when the job has completed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4088) Task stuck in JobLocalizer prevented other tasks on the same node from committing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263246#comment-13263246 ] Ravi Prakash commented on MAPREDUCE-4088: - Does anyone have any suggestions on how to fix this? I'm thinking of this: * We could have a timeout in the wait for checkJobStatusAndWait. * If we time out, we simply put the action back into the queue (hoping next time around it succeeds) * This might make the isIdle method more complicated :( Task stuck in JobLocalizer prevented other tasks on the same node from committing - Key: MAPREDUCE-4088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4088 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.20.205.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Critical We saw that as a result of HADOOP-6963, one task was stuck in this Thread 23668: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=3, line=446 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) TONS MORE OF THIS SAME LINE - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) . . - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Interpreted frame) ne=451 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects(org.apache.hadoop.conf.Configuration, java.net.URI[], org.apache.hadoop.fs.Path[], long[], boolean[], boolean) @bci=150, line=324 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCache(org.apache.hadoop.conf.Configuration) @bci=40, line=349 (Interpreted frame) 51, line=383 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.runSetup(java.lang.String, java.lang.String, org.apache.hadoop.fs.Path, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=46, line=477 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=20, line=534 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=1, line=531 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=14, line=1082 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.main(java.lang.String[]) @bci=266, line=530 (Interpreted frame) While all other tasks on the same node were stuck in Thread 32141: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) - org.apache.hadoop.mapred.Task.commit(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter, org.apache.hadoop.mapreduce.OutputCommitter) @bci=24, line=980 (Compiled frame) - org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter) @bci=146, line=871 (Interpreted frame) - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=470, line=423 (Interpreted frame) - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=255 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=14, line=1082 (Interpreted frame) - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=738, line=249 (Interpreted frame) This should never happen. A stuck task should never prevent other tasks from different jobs on the same node from committing. -- This message is automatically generated by JIRA. If you think it was sent
[jira] [Updated] (MAPREDUCE-4204) Refactor ProcfsBasedProcessTree to make the resource collection object pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-4204: -- Attachment: MAPREDUCE-4204.patch Refactor patch. Replaces ProcfsBasedProcessTree with an abstract class ResourceCalculatorProcessTree. Refactor ProcfsBasedProcessTree to make the resource collection object pluggable Key: MAPREDUCE-4204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4204 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Bikas Saha Assignee: Bikas Saha Attachments: MAPREDUCE-4204.patch Making it a pluggable interface will allow replacing the procfs based implementation with ones for other platforms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3873) Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263322#comment-13263322 ] xieguiming commented on MAPREDUCE-3873: --- The modified function: public boolean isValidNode(String hostName) { synchronized (hostsReader) { SetString hostsList = hostsReader.getHosts(); SetString excludeList = hostsReader.getExcludedHosts(); {color:red} String ip = NetUtils.normalizeHostName(hostName); {color} return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList .contains(ip)) !(excludeList.contains(hostName) || excludeList.contains(ip)); } } and the function NetUtils.normalizeHostName(hostName) can not be mocked. So, I think this patch do not need provide the unit testcase. Nodemanager is not getting decommisioned if the absolute ip is given in exclude file. - Key: MAPREDUCE-3873 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3873 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Nishan Shetty Priority: Minor Attachments: MAPREDUCE-3873.patch Configure absolute ip in yarn.resourcemanager.nodes.exclude-path and try to decommission the node. It is not getting decommisioned.But if the hostname is given, decommissioning is happening. I have also given the ip-host mapping of each machine in /etc/hosts.(i,e in every machine the other machines ip-host mapping is specified). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4052) Windows eclpise can not submit the job
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xieguiming reassigned MAPREDUCE-4052: - Assignee: xieguiming Windows eclpise can not submit the job -- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: xieguiming Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3873) Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xieguiming reassigned MAPREDUCE-3873: - Assignee: xieguiming Nodemanager is not getting decommisioned if the absolute ip is given in exclude file. - Key: MAPREDUCE-3873 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3873 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Nishan Shetty Assignee: xieguiming Priority: Minor Attachments: MAPREDUCE-3873.patch Configure absolute ip in yarn.resourcemanager.nodes.exclude-path and try to decommission the node. It is not getting decommisioned.But if the hostname is given, decommissioning is happening. I have also given the ip-host mapping of each machine in /etc/hosts.(i,e in every machine the other machines ip-host mapping is specified). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4088) Task stuck in JobLocalizer prevented other tasks on the same node from committing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-4088: Attachment: MAPREDUCE-4088.patch I'm thinking of a patch like this. If you have any concerns, speak now. Or forever hold thy peace! Task stuck in JobLocalizer prevented other tasks on the same node from committing - Key: MAPREDUCE-4088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4088 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.20.205.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-4088.patch We saw that as a result of HADOOP-6963, one task was stuck in this Thread 23668: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=3, line=446 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) TONS MORE OF THIS SAME LINE - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) . . - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Interpreted frame) ne=451 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects(org.apache.hadoop.conf.Configuration, java.net.URI[], org.apache.hadoop.fs.Path[], long[], boolean[], boolean) @bci=150, line=324 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCache(org.apache.hadoop.conf.Configuration) @bci=40, line=349 (Interpreted frame) 51, line=383 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.runSetup(java.lang.String, java.lang.String, org.apache.hadoop.fs.Path, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=46, line=477 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=20, line=534 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=1, line=531 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=14, line=1082 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.main(java.lang.String[]) @bci=266, line=530 (Interpreted frame) While all other tasks on the same node were stuck in Thread 32141: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) - org.apache.hadoop.mapred.Task.commit(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter, org.apache.hadoop.mapreduce.OutputCommitter) @bci=24, line=980 (Compiled frame) - org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter) @bci=146, line=871 (Interpreted frame) - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=470, line=423 (Interpreted frame) - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=255 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=14, line=1082 (Interpreted frame) - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=738, line=249 (Interpreted frame) This should never happen. A stuck task should never prevent other tasks from different jobs on the same node from committing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: