date:20120426


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xieguiming updated MAPREDUCE-3873:
--

Attachment: MAPREDUCE-3873.patch

 Nodemanager is not getting decommisioned if the absolute ip is given in 
 exclude file.
 -

 Key: MAPREDUCE-3873
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3873
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Nishan Shetty
Priority: Minor
 Attachments: MAPREDUCE-3873.patch


 Configure absolute ip in yarn.resourcemanager.nodes.exclude-path and try to 
 decommission the node.
 It is not getting decommisioned.But if the hostname is given, decommissioning 
 is happening.
 I have also given the ip-host mapping of each machine in /etc/hosts.(i,e in 
 every machine the other machines ip-host mapping is specified).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3873) Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xieguiming updated MAPREDUCE-3873:
--

Status: Patch Available  (was: Open)

The patch will support that we config either hostname or ip in the 
yarn.resourcemanager.nodes.exclude-path and 
yarn.resourcemanager.nodes.include-path file.

 Nodemanager is not getting decommisioned if the absolute ip is given in 
 exclude file.
 -

 Key: MAPREDUCE-3873
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3873
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Nishan Shetty
Priority: Minor
 Attachments: MAPREDUCE-3873.patch


 Configure absolute ip in yarn.resourcemanager.nodes.exclude-path and try to 
 decommission the node.
 It is not getting decommisioned.But if the hostname is given, decommissioning 
 is happening.
 I have also given the ip-host mapping of each machine in /etc/hosts.(i,e in 
 every machine the other machines ip-host mapping is specified).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262571#comment-13262571
 ] 

Hudson commented on MAPREDUCE-4194:
---

Integrated in Hadoop-Hdfs-0.23-Build #239 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/239/])
svn merge -c 1330552 FIXES: MAPREDUCE-4194. ConcurrentModificationError in 
DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330556)

 Result = UNSTABLE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330556
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java


 ConcurrentModificationError in DirectoryCollection
 --

 Key: MAPREDUCE-4194
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
 Fix For: 0.23.3, 2.0.0, 3.0.0

 Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch


 As found as part of work on MAPREDUCE-4169, it is possible for a 
 ConcurrentModificationException to be thrown upon disk failure. 
 DirectoryCollection hands out its internal list structure that is accessed 
 across multiple threads. Upon disk failure its internal list is modified, 
 invalidating all current iterators to that structure.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262570#comment-13262570
 ] 

Hudson commented on MAPREDUCE-3613:
---

Integrated in Hadoop-Hdfs-0.23-Build #239 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/239/])
merge -r 1330560:1330561 from branch-2. FIXES: MAPREDUCE-3613 (Revision 
1330562)

 Result = UNSTABLE
tgraves : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330562
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java


 web service calls header contains 2 content types
 -

 Key: MAPREDUCE-3613
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.0
Reporter: Thomas Graves
Assignee: Thomas Graves
Priority: Critical
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3613.patch


 when doing requesting info from the web services rest API, curl seems to 
 return content-type of both text and json or xml:
  Accept: application/xml
 
  HTTP/1.1 200 OK
  Content-Type: text/plain; charset=utf-8
  Content-Type: application/xml

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4199) Not easy to search the property because of splited into many parts.

xieguiming created MAPREDUCE-4199:
-

 Summary: Not easy to search the property  because of splited into 
many parts.
 Key: MAPREDUCE-4199
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4199
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 0.23.1
 Environment: suse
Reporter: xieguiming
Priority: Minor


The property is splited into many parts. such as:

public static final String RM_NODES_EXCLUDE_FILE_PATH = 
RM_PREFIX + nodes.exclude-path;

It is inconvenient for code reader to search the code by the property name 
although there are some common parts for reuse.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262606#comment-13262606
 ] 

Hudson commented on MAPREDUCE-4193:
---

Integrated in Hadoop-Hdfs-trunk #1026 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/])
MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml. 
Contributed by Patrick Hunt. (Revision 1330427)

 Result = FAILURE
tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330427
Files : 
* /hadoop/common/trunk/BUILDING.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl/configuration.xsl
* /hadoop/common/trunk/hadoop-project/src/site/site.xml


 broken doc link for yarn-default.xml in site.xml
 

 Key: MAPREDUCE-4193
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: documentation
Reporter: Patrick Hunt
Assignee: Patrick Hunt
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4193.patch


 the link to yarn-default.xml in site.xml is incorrect, generated docs link is 
 broken.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262605#comment-13262605
 ] 

Hudson commented on MAPREDUCE-3613:
---

Integrated in Hadoop-Hdfs-trunk #1026 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/])
MAPREDUCE-3613. web service calls header contains 2 content types (tgraves) 
(Revision 1330560)

 Result = FAILURE
tgraves : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330560
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java


 web service calls header contains 2 content types
 -

 Key: MAPREDUCE-3613
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.0
Reporter: Thomas Graves
Assignee: Thomas Graves
Priority: Critical
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3613.patch


 when doing requesting info from the web services rest API, curl seems to 
 return content-type of both text and json or xml:
  Accept: application/xml
 
  HTTP/1.1 200 OK
  Content-Type: text/plain; charset=utf-8
  Content-Type: application/xml

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262607#comment-13262607
 ] 

Hudson commented on MAPREDUCE-4194:
---

Integrated in Hadoop-Hdfs-trunk #1026 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1026/])
MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection 
(Jonathan Eagles via bobby) (Revision 1330552)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330552
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java


 ConcurrentModificationError in DirectoryCollection
 --

 Key: MAPREDUCE-4194
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
 Fix For: 0.23.3, 2.0.0, 3.0.0

 Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch


 As found as part of work on MAPREDUCE-4169, it is possible for a 
 ConcurrentModificationException to be thrown upon disk failure. 
 DirectoryCollection hands out its internal list structure that is accessed 
 across multiple threads. Upon disk failure its internal list is modified, 
 invalidating all current iterators to that structure.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262619#comment-13262619
 ] 

Hudson commented on MAPREDUCE-3613:
---

Integrated in Hadoop-Mapreduce-trunk #1061 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/])
MAPREDUCE-3613. web service calls header contains 2 content types (tgraves) 
(Revision 1330560)

 Result = FAILURE
tgraves : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330560
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java


 web service calls header contains 2 content types
 -

 Key: MAPREDUCE-3613
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.0
Reporter: Thomas Graves
Assignee: Thomas Graves
Priority: Critical
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3613.patch


 when doing requesting info from the web services rest API, curl seems to 
 return content-type of both text and json or xml:
  Accept: application/xml
 
  HTTP/1.1 200 OK
  Content-Type: text/plain; charset=utf-8
  Content-Type: application/xml

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262620#comment-13262620
 ] 

Hudson commented on MAPREDUCE-4193:
---

Integrated in Hadoop-Mapreduce-trunk #1061 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/])
MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml. 
Contributed by Patrick Hunt. (Revision 1330427)

 Result = FAILURE
tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330427
Files : 
* /hadoop/common/trunk/BUILDING.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl/configuration.xsl
* /hadoop/common/trunk/hadoop-project/src/site/site.xml


 broken doc link for yarn-default.xml in site.xml
 

 Key: MAPREDUCE-4193
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: documentation
Reporter: Patrick Hunt
Assignee: Patrick Hunt
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4193.patch


 the link to yarn-default.xml in site.xml is incorrect, generated docs link is 
 broken.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262621#comment-13262621
 ] 

Hudson commented on MAPREDUCE-4194:
---

Integrated in Hadoop-Mapreduce-trunk #1061 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1061/])
MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection 
(Jonathan Eagles via bobby) (Revision 1330552)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1330552
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java


 ConcurrentModificationError in DirectoryCollection
 --

 Key: MAPREDUCE-4194
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
 Fix For: 0.23.3, 2.0.0, 3.0.0

 Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch


 As found as part of work on MAPREDUCE-4169, it is possible for a 
 ConcurrentModificationException to be thrown upon disk failure. 
 DirectoryCollection hands out its internal list structure that is accessed 
 across multiple threads. Upon disk failure its internal list is modified, 
 invalidating all current iterators to that structure.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-4191) capacity scheduler: job unexpectedly exceeds queue capacity limit by one task

2012-04-26 Thread Thomas Graves (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves reassigned MAPREDUCE-4191:


Assignee: Thomas Graves

 capacity scheduler: job unexpectedly exceeds queue capacity limit by one task
 -

 Key: MAPREDUCE-4191
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4191
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, scheduler
Affects Versions: 0.23.3
Reporter: Thomas Graves
Assignee: Thomas Graves

 While testing the queue capacity limits, it appears that the job can exceed 
 the
 queue capacity limit by one task while the user limit factor is 1. It's not
 clear to me why this is. 
 Here is the steps to reproduce:
 1) set yarn.app.mapreduce.am.resource.mb to 2048 (default value)
 2) set yarn.scheduler.capacity.root.default.user-limit-factor to 1.0 (default)
 3) set yarn.scheduler.capacity.root.default.capacity to 90 (%)
 4) For a cluster with capacity of 56G, 90% rounded up is 51.
 5) submit a job with large number of tasks, each task using 1G memory. 
 6) webui shows that the used resource is 52 G, which is 92.9% of the cluster
 capacity (instead of the expected 90%), and 103.2% of the queue capacity
 (instead of the expected 100%).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3873) Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.

[
https://issues.apache.org/jira/browse/MAPREDUCE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262623#comment-13262623
]

Hadoop QA commented on MAPREDUCE-3873:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524425/MAPREDUCE-3873.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity

org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens

org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService

org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry

org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider
org.apache.hadoop.mapreduce.security.TestJHSSecurity

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2314//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2314//console

This message is automatically generated.

Nodemanager is not getting decommisioned if the absolute ip is given in
exclude file.
-

Key: MAPREDUCE-3873
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3873
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Nishan Shetty
Priority: Minor
Attachments: MAPREDUCE-3873.patch

Configure absolute ip in yarn.resourcemanager.nodes.exclude-path and try to
decommission the node.
It is not getting decommisioned.But if the hostname is given, decommissioning
is happening.
I have also given the ip-host mapping of each machine in /etc/hosts.(i,e in
every machine the other machines ip-host mapping is specified).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4155) Provide support for these yarn commands

2012-04-26 Thread Devaraj K (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4155:
-

Attachment: MAPREDUCE-4155.patch

 Provide support for these yarn commands
 ---

 Key: MAPREDUCE-4155
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4155
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0
Reporter: Devaraj K
Assignee: Devaraj K
 Attachments: MAPREDUCE-4155.patch


 1. status app-id
 2. kill app-id (Already issue present with Id : MAPREDUCE-3793)
 3. list-apps [all]
 4. nodes-report

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4155) Provide support for these yarn commands

2012-04-26 Thread Devaraj K (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4155:
-

Affects Version/s: 3.0.0
   Status: Patch Available  (was: Open)

I have attached patch for this. Please review and give your comments on this.

 Provide support for these yarn commands
 ---

 Key: MAPREDUCE-4155
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4155
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
 Attachments: MAPREDUCE-4155.patch


 1. status app-id
 2. kill app-id (Already issue present with Id : MAPREDUCE-3793)
 3. list-apps [all]
 4. nodes-report

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4200) packaging tar ball of trunk failed

Tomohiko Kinebuchi created MAPREDUCE-4200:
-

 Summary: packaging tar ball of trunk failed
 Key: MAPREDUCE-4200
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: trunk
 Environment: Mac OS X Lion 10.7.3
Reporter: Tomohiko Kinebuchi


A command mvn clean package -Dtar -DskipTests executed on the root directory 
hadoop-common failed.
Its output logs are in an attached file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4200) packaging tar ball of trunk failed


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomohiko Kinebuchi updated MAPREDUCE-4200:
--

Attachment: MAPREDUCE-4200.error.log

 packaging tar ball of trunk failed
 --

 Key: MAPREDUCE-4200
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: trunk
 Environment: Mac OS X Lion 10.7.3
Reporter: Tomohiko Kinebuchi
 Attachments: MAPREDUCE-4200.error.log


 A command mvn clean package -Dtar -DskipTests executed on the root 
 directory hadoop-common failed.
 Its output logs are in an attached file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4155) Provide support for these yarn commands

[
https://issues.apache.org/jira/browse/MAPREDUCE-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262667#comment-13262667
]

Hadoop QA commented on MAPREDUCE-4155:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524439/MAPREDUCE-4155.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity

org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens

org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService

org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry

org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2315//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2315//console

This message is automatically generated.

Provide support for these yarn commands
---

Key: MAPREDUCE-4155
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4155
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4155.patch

1. status app-id
2. kill app-id (Already issue present with Id : MAPREDUCE-3793)
3. list-apps [all]
4. nodes-report

[jira] [Commented] (MAPREDUCE-4200) packaging tar ball of trunk failed

2012-04-26 Thread Radim Kolar (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262688#comment-13262688
 ] 

Radim Kolar commented on MAPREDUCE-4200:


did you looked at builds.apache.org if your error is Mac OS specific?

 packaging tar ball of trunk failed
 --

 Key: MAPREDUCE-4200
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: trunk
 Environment: Mac OS X Lion 10.7.3
Reporter: Tomohiko Kinebuchi
 Attachments: MAPREDUCE-4200.error.log


 A command mvn clean package -Dtar -DskipTests executed on the root 
 directory hadoop-common failed.
 Its output logs are in an attached file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4200) packaging tar ball of trunk failed


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262693#comment-13262693
 ] 

Tomohiko Kinebuchi commented on MAPREDUCE-4200:
---

@Radim

Thanks for your comment.

Results of the last five builds are all failed.
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/
So this issue is not Mac OS specific.

 packaging tar ball of trunk failed
 --

 Key: MAPREDUCE-4200
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: trunk
 Environment: Mac OS X Lion 10.7.3
Reporter: Tomohiko Kinebuchi
 Attachments: MAPREDUCE-4200.error.log


 A command mvn clean package -Dtar -DskipTests executed on the root 
 directory hadoop-common failed.
 Its output logs are in an attached file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4200) packaging tar ball of trunk failed


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomohiko Kinebuchi updated MAPREDUCE-4200:
--

Description: 
A command mvn clean package -Dtar -DskipTests executed on the root directory 
hadoop-common failed.
Its output logs are in an attached file.

A command mvn clean package -Pdist -DskipTests succeeded.


  was:
A command mvn clean package -Dtar -DskipTests executed on the root directory 
hadoop-common failed.
Its output logs are in an attached file.


 packaging tar ball of trunk failed
 --

 Key: MAPREDUCE-4200
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: trunk
 Environment: Mac OS X Lion 10.7.3
Reporter: Tomohiko Kinebuchi
 Attachments: MAPREDUCE-4200.error.log


 A command mvn clean package -Dtar -DskipTests executed on the root 
 directory hadoop-common failed.
 Its output logs are in an attached file.
 A command mvn clean package -Pdist -DskipTests succeeded.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4200) packaging tar ball of trunk failed


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262751#comment-13262751
 ] 

Tomohiko Kinebuchi commented on MAPREDUCE-4200:
---

From output messages, this ant target executed and failed.

https://github.com/apache/hadoop-common/blob/235bcba59b02f8a3234d042010cf74f8dcf4a2fa/hadoop-mapreduce-project/pom.xml#L178

 packaging tar ball of trunk failed
 --

 Key: MAPREDUCE-4200
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4200
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build
Affects Versions: trunk
 Environment: Mac OS X Lion 10.7.3
Reporter: Tomohiko Kinebuchi
 Attachments: MAPREDUCE-4200.error.log


 A command mvn clean package -Dtar -DskipTests executed on the root 
 directory hadoop-common failed.
 Its output logs are in an attached file.
 A command mvn clean package -Pdist -DskipTests succeeded.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4191) capacity scheduler: job unexpectedly exceeds queue capacity limit by one task

2012-04-26 Thread Thomas Graves (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262799#comment-13262799
]

Thomas Graves commented on MAPREDUCE-4191:
--

I'm still following this through to fully understand, but there is a comment in
the code in LeafQueue that tries to explain this:

// Note: We aren't considering the current request since there is a fixed
// overhead of the AM, but it's a check, not a = check, so...

Which I don't totally follow, I guess if you have one job in the queue that is
taking the entire capacity, it allows the job to be more like it was in mrv1
and tries not to penalize you for the AM overhead. The AM however is doing the
setup and clean tasks where as in mrv1 it would need to allocate a slot for
those. The AM may have fixed overhead but that overhead is configurable. I
could create an AM with 24G of memory or use the default of 1.5G. Or on the
flip side, I have an AM that uses 1.5G, but have a map task that now gets
scheduled and uses 24G which puts it way over its capacity. That could affect
the queue current usage greatly and seems to break the capacity guarantee.

In the case where you say have 2 jobs in the queue, you have 2 app masters, one
of which is counted' against your queue and then the other one is not.

I do see it beneficial to queues with very small capacities though, as without
this they could be stuck without enough resources to run a task.

Arun or anyone else familiar with capacity scheduler, if you could provide
explanation that would be great.

capacity scheduler: job unexpectedly exceeds queue capacity limit by one task
-

Key: MAPREDUCE-4191
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4191
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, scheduler
Affects Versions: 0.23.3
Reporter: Thomas Graves
Assignee: Thomas Graves

While testing the queue capacity limits, it appears that the job can exceed
the
queue capacity limit by one task while the user limit factor is 1. It's not
clear to me why this is.
Here is the steps to reproduce:
1) set yarn.app.mapreduce.am.resource.mb to 2048 (default value)
2) set yarn.scheduler.capacity.root.default.user-limit-factor to 1.0 (default)
3) set yarn.scheduler.capacity.root.default.capacity to 90 (%)
4) For a cluster with capacity of 56G, 90% rounded up is 51.
5) submit a job with large number of tasks, each task using 1G memory.
6) webui shows that the used resource is 52 G, which is 92.9% of the cluster
capacity (instead of the expected 90%), and 103.2% of the queue capacity
(instead of the expected 100%).

[jira] [Updated] (MAPREDUCE-4066) To get yarn.app.mapreduce.am.staging-dir value, should set the default value

2012-04-26 Thread Tsz Wo (Nicholas), SZE (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated MAPREDUCE-4066:
--

Assignee: xieguiming

 To get yarn.app.mapreduce.am.staging-dir value, should set the default value
 --

 Key: MAPREDUCE-4066
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4066
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: job submission, mrv2
Affects Versions: 0.23.1
 Environment: client is windows eclipse, server is suse
Reporter: xieguiming
Assignee: xieguiming
Priority: Minor
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4066.patch, MAPREDUCE-4066.patch


 when submit the job use the windows eclipse, and the 
 yarn.app.mapreduce.am.staging-dir value is null.
 {code:title=MRApps.java|borderStyle=solid}
   public static Path getStagingAreaDir(Configuration conf, String user) {
 return new Path(
 conf.get(MRJobConfig.MR_AM_STAGING_DIR) + 
 Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT);
   }
 {code}
 should modify to:
 {code:title=MRApps.java|borderStyle=solid}
   public static Path getStagingAreaDir(Configuration conf, String user) {
 return new Path(
 conf.get(MRJobConfig.MR_AM_STAGING_DIR,/tmp/hadoop-yarn/staging) + 
 Path.SEPARATOR + user + Path.SEPARATOR + STAGING_CONSTANT);
   }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4074) Client continuously retries to RM When RM goes down before launching Application Master

2012-04-26 Thread Tsz Wo (Nicholas), SZE (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE updated MAPREDUCE-4074:
--

Assignee: xieguiming

 Client continuously retries to RM When RM goes down before launching 
 Application Master
 ---

 Key: MAPREDUCE-4074
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4074
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: xieguiming
 Fix For: 0.23.3

 Attachments: MAPREDUCE-4074-1.patch, MAPREDUCE-4074-2.patch, 
 MAPREDUCE-4074-3.patch, MAPREDUCE-4074.patch


 Client continuously tries to RM and logs the below messages when the RM goes 
 down before launching App Master. 
 I feel exception should be thrown or break the loop after finite no of 
 retries.
 {code:xml}
 28/03/12 07:15:03 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
 28/03/12 07:15:04 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
 28/03/12 07:15:05 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
 28/03/12 07:15:06 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
 28/03/12 07:15:07 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
 28/03/12 07:15:08 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 5 time(s).
 28/03/12 07:15:09 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 6 time(s).
 28/03/12 07:15:10 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 7 time(s).
 28/03/12 07:15:11 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 8 time(s).
 28/03/12 07:15:12 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 9 time(s).
 28/03/12 07:15:13 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
 28/03/12 07:15:14 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
 28/03/12 07:15:15 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
 28/03/12 07:15:16 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
 28/03/12 07:15:17 INFO ipc.Client: Retrying connect to server: 
 linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3173) MRV2 UI doesn't work properly without internet

[
https://issues.apache.org/jira/browse/MAPREDUCE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262819#comment-13262819
]

Robert Joseph Evans commented on MAPREDUCE-3173:

I tried to to set this up, but I am getting some 404 errors for images from the
base theme. I don't know if they are critical or not, but I have to assume
that they are, and that other themes are missing images too. Could you provide
a tgz of what you have under
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/jquery/themes-1.8.16/
instead of the patch as that way we can get all of the images too.

MRV2 UI doesn't work properly without internet
--

Key: MAPREDUCE-3173
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3173
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-3173.patch, jquery-ui.min-1.8.16.js.gz,
jquery.min-1.6.4.js.gz

When we try access the MRV2 UI, it is always giving the below message in the
UI even if the java script enabled in the browser.
{code:xml}
This page works best with javascript enabled.
{code}
It is trying to download these below css/js files from internet and finally
ending up with the above message. For loading the page also it is taking long
time.
{code:title=JQueryUI.java|borderStyle=solid}
html.

link(join(https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/themes/;,
getTheme(), /jquery-ui.css)).
link(/static/dt-1.7.5/css/jui-dt.css).

script(https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js;).

script(https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/jquery-ui.min.js;).
{code}

[jira] [Updated] (MAPREDUCE-3173) MRV2 UI doesn't work properly without internet


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-3173:
---

Status: Open  (was: Patch Available)

 MRV2 UI doesn't work properly without internet
 --

 Key: MAPREDUCE-3173
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3173
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
 Attachments: MAPREDUCE-3173.patch, jquery-ui.min-1.8.16.js.gz, 
 jquery.min-1.6.4.js.gz


 When we try access the MRV2 UI, it is always giving the below message in the 
 UI even if the java script enabled in the browser.
 {code:xml}
 This page works best with javascript enabled. 
 {code}
 It is trying to download these below css/js files from internet and finally 
 ending up with the above message. For loading the page also it is taking long 
 time.
 {code:title=JQueryUI.java|borderStyle=solid}
   html.
   
 link(join(https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/themes/;,
 getTheme(), /jquery-ui.css)).
   link(/static/dt-1.7.5/css/jui-dt.css).
   
 script(https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js;).
   
 script(https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/jquery-ui.min.js;).
 {code} 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4169) Container Logs appear in unsorted order


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4169:
---

  Resolution: Fixed
   Fix Version/s: 3.0.0
  2.0.0
  0.23.3
Target Version/s: 0.23.3, 2.0.0  (was: 2.0.0, 0.23.3)
  Status: Resolved  (was: Patch Available)

Thanks John,

I put this into trunk, branch-2, and branch-0.23

 Container Logs appear in unsorted order
 ---

 Key: MAPREDUCE-4169
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Minor
 Fix For: 0.23.3, 2.0.0, 3.0.0

 Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, 
 MAPREDUCE-4169.patch


 container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory 
 ui appear in unsorted order where the order displayed is based on what file 
 was created first. This jira will have the results be displayed in a 
 consistent order.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4138) Reduce memory usage of counters due to non-static nested classes

2012-04-26 Thread Tom White (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom White updated MAPREDUCE-4138:
-

   Resolution: Fixed
Fix Version/s: 2.0.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

I just committed this. Thanks for reviewing, Alejandro and Robert.

 Reduce memory usage of counters due to non-static nested classes
 

 Key: MAPREDUCE-4138
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4138
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Tom White
Assignee: Tom White
 Fix For: 2.0.0

 Attachments: MAPREDUCE-4138.patch, MAPREDUCE-4138.patch


 FrameworkCounter is a non-static nested class of FrameworkCounterGroup which 
 means it retains a reference to the outer class, which isn't really needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order

2012-04-26 Thread Jonathan Eagles (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13262894#comment-13262894
 ] 

Jonathan Eagles commented on MAPREDUCE-4169:


Awesome, thanks so much for the thorough reviews Bobby and Harsh.

Jon

 Container Logs appear in unsorted order
 ---

 Key: MAPREDUCE-4169
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Priority: Minor
 Fix For: 0.23.3, 2.0.0, 3.0.0

 Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, 
 MAPREDUCE-4169.patch


 container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory 
 ui appear in unsorted order where the order displayed is based on what file 
 was created first. This jira will have the results be displayed in a 
 consistent order.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4189) TestContainerManagerSecurity is failing


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4189:
---

   Resolution: Fixed
Fix Version/s: 3.0.0
   2.0.0
   0.23.3
   Status: Resolved  (was: Patch Available)

Thanks Devaraj,

I have put this into trunk, branch-2, and branch-0.23.

 TestContainerManagerSecurity is failing
 ---

 Key: MAPREDUCE-4189
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: trunk
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Critical
 Fix For: 0.23.3, 2.0.0, 3.0.0

 Attachments: MAPREDUCE-4189.patch, MAPREDUCE-4189.patch


 {code:xml}
 ---
  T E S T S
 ---
 Running org.apache.hadoop.yarn.server.TestDiskFailures
 Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
 Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
 Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec 
  FAILURE!
 Results :
 Tests in error:
   
 testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
   testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
   
 testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
 Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4201) Getting PID not working on Windows. Termination of Task/TaskJVM's not working

Bikas Saha created MAPREDUCE-4201:
-

 Summary: Getting PID not working on Windows. Termination of 
Task/TaskJVM's not working
 Key: MAPREDUCE-4201
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4201
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Bikas Saha
Assignee: Bikas Saha


Child Task not reporting PID because of Linux specific shell script 
implementation.
Signaling task termination currently disabled by the initial Windows patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4155) Provide support for these yarn commands

2012-04-26 Thread Ahmed Radwan (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263114#comment-13263114
 ] 

Ahmed Radwan commented on MAPREDUCE-4155:
-

Thanks Devaraj for this addition! +1, can you please add docs for these new 
commands (e.g. in YarnCommands.apt.vm).

 Provide support for these yarn commands
 ---

 Key: MAPREDUCE-4155
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4155
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
 Attachments: MAPREDUCE-4155.patch


 1. status app-id
 2. kill app-id (Already issue present with Id : MAPREDUCE-3793)
 3. list-apps [all]
 4. nodes-report

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4201) Getting PID not working on Windows. Termination of Task/TaskJVM's not working


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated MAPREDUCE-4201:
--

Attachment: MAPREDUCE-4201.patch

patch addresses the issues.
Using windows built in tasklist and taskkill commands for the process 
management in util.ProcessTree.
Plan to write equivalent of ProcFsBasedProcessTree later on and add equivalent 
tests.

 Getting PID not working on Windows. Termination of Task/TaskJVM's not working
 -

 Key: MAPREDUCE-4201
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4201
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: MAPREDUCE-4201.patch


 Child Task not reporting PID because of Linux specific shell script 
 implementation.
 Signaling task termination currently disabled by the initial Windows patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4202) TestYarnClientProtocolProvider is broken

2012-04-26 Thread Daryn Sharp (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated MAPREDUCE-4202:
---

Attachment: MAPREDUCE-4202.patch

Use same type of conditional as yarn's protocol provider.  The test and related 
files haven't been touched in a long time, so I have no idea how this test ever 
passed unless the CLASSPATH changed such that the service provider started 
finding the local provider class.

 TestYarnClientProtocolProvider is broken
 

 Key: MAPREDUCE-4202
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4202
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Affects Versions: 0.24.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
 Attachments: MAPREDUCE-4202.patch


 The test fails because a cluster is unexpectedly created with an empty conf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-3987) Reduce RM app memory footprint once app has completed

2012-04-26 Thread Jonathan Eagles (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles reassigned MAPREDUCE-3987:
--

Assignee: (was: Jonathan Eagles)

 Reduce RM app memory footprint once app has completed
 -

 Key: MAPREDUCE-3987
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3987
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2, resourcemanager
Affects Versions: 0.23.1
Reporter: Jason Lowe

 The ResourceManager holds onto a configurable number of completed 
 applications (yarn.resource.max-completed-applications, defaults to 1), 
 and the memory footprint of these completed applications can be significant.  
 For example, the {{submissionContext}} in RMAppImpl contains references to 
 protocolbuffer objects and other items that probably aren't necessary to keep 
 around once the application has completed.  We could significantly reduce the 
 memory footprint of the RM by releasing objects that are no longer necessary 
 once an application completes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4002) MultiFileWordCount job fails if the input path is not from default file system

2012-04-26 Thread Ahmed Radwan (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263155#comment-13263155
 ] 

Ahmed Radwan commented on MAPREDUCE-4002:
-

+1, change lgtm. Thanks!

 MultiFileWordCount job fails if the input path is not from default file system
 --

 Key: MAPREDUCE-4002
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4002
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: examples
Affects Versions: 0.24.0, 0.23.3
Reporter: Bhallamudi Venkata Siva Kamesh
 Attachments: MAPREDUCE-4002.patch


 In the MultiFileWordCount#CombineFileLineRecordReader, filesystem object has 
 been initialized in the following way
 {noformat}fs = FileSystem.get(context.getConfiguration());{noformat}
 This causes, *fs* to be initialized with default filesystem. Therefore *fs* 
 searchs for the input files on the default file system, which fails if the 
 input path is from different source.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3871) Allow symlinking in LocalJobRunner DistributedCache

2012-04-26 Thread Tom White (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom White updated MAPREDUCE-3871:
-

Attachment: MAPREDUCE-3871.patch

Small change to add a negative test for files that should not be symlinked 
(check that no symlink is present).

 Allow symlinking in LocalJobRunner DistributedCache
 ---

 Key: MAPREDUCE-3871
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3871
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: distributed-cache
Reporter: Tom White
Assignee: Tom White
 Attachments: MAPREDUCE-3871.patch, MAPREDUCE-3871.patch


 Currently the LocalJobRunner doesn't create symlinks for files in the 
 DistributedCache. It is safe to create symlinks if files of the same name 
 don't exist. LocalJobRunner should also delete the symlinks when the job has 
 completed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4088) Task stuck in JobLocalizer prevented other tasks on the same node from committing

2012-04-26 Thread Ravi Prakash (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263197#comment-13263197
 ] 

Ravi Prakash commented on MAPREDUCE-4088:
-

Courtesy Koji

{quote}
Each TaskTracker has a single thread for taskCleanup taking work from
tasksToCleanup queue.

For each task to cleanup, it firsts call 
  checkJobStatusAndWait(action);

And inside
 {noformat}
  private void checkJobStatusAndWait(TaskTrackerAction action)
...
synchronized (runningJobs) {
  rjob = runningJobs.get(jobId);
}
if (rjob != null) {
  synchronized (rjob) {
while (rjob.localizing) {
  rjob.wait();
}
  }
}
 {noformat}

So this thread would wait while the task is being localized.
Even if one task is hung on localization, entire cleanup is stopped.
{quote}

East or west! Koji is the best!
Soda lemon ginger pop! Koji is on the top!
Yyaayyy yaa yaa for Koji!

 Task stuck in JobLocalizer prevented other tasks on the same node from 
 committing
 -

 Key: MAPREDUCE-4088
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4088
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1
Affects Versions: 0.20.205.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Priority: Critical

 We saw that as a result of HADOOP-6963, one task was stuck in this
 Thread 23668: (state = IN_NATIVE)
  - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 
 (Compiled frame; information may be imprecise)
  - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 
 (Compiled frame)
  - java.io.File.exists() @bci=20, line=733 (Compiled frame)
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=3, line=446 
 (Compiled frame)
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Compiled frame)
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Compiled frame)
 
  TONS MORE OF THIS SAME LINE
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Compiled frame)
 .
 .
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Compiled frame)
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Interpreted frame)
 ne=451 (Interpreted frame)
  - 
 org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects(org.apache.hadoop.conf.Configuration,
  java.net.URI[], org.apache.hadoop.fs.Path[], long[], boolean[], boolean) 
 @bci=150, line=324 (Interpreted frame)
  - 
 org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCache(org.apache.hadoop.conf.Configuration)
  @bci=40, line=349 (Interpreted frame) 51, line=383 (Interpreted frame)
  - org.apache.hadoop.mapred.JobLocalizer.runSetup(java.lang.String, 
 java.lang.String, org.apache.hadoop.fs.Path, 
 org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=46, line=477 
 (Interpreted frame)
  - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=20, line=534 
 (Interpreted frame)
  - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=1, line=531 
 (Interpreted frame)
  - 
 java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
  java.security.AccessControlContext) @bci=0 (Interpreted frame)
  - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
 java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame)
  - 
 org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
  @bci=14, line=1082 (Interpreted frame)
  - org.apache.hadoop.mapred.JobLocalizer.main(java.lang.String[]) @bci=266, 
 line=530 (Interpreted frame)
 While all other tasks on the same node were stuck in 
 Thread 32141: (state = BLOCKED)
  - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
  - 
 org.apache.hadoop.mapred.Task.commit(org.apache.hadoop.mapred.TaskUmbilicalProtocol,
  org.apache.hadoop.mapred.Task$TaskReporter, 
 org.apache.hadoop.mapreduce.OutputCommitter) @bci=24, line=980 (Compiled 
 frame)
  - 
 org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol,
  org.apache.hadoop.mapred.Task$TaskReporter) @bci=146, line=871 (Interpreted 
 frame)
  - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf, 
 org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=470, line=423 
 (Interpreted frame)
  - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=255 (Interpreted 
 frame)
  - 
 java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
  java.security.AccessControlContext) @bci=0 (Interpreted frame)
  - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
 java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame)
  -

[jira] [Created] (MAPREDUCE-4203) Create equivalent of ProcfsBasedProcessTree for Windows

Bikas Saha created MAPREDUCE-4203:
-

 Summary: Create equivalent of ProcfsBasedProcessTree for Windows
 Key: MAPREDUCE-4203
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha


ProcfsBasedProcessTree is used by the TaskTracker to get process information 
like memory and cpu usage. This information is used to manage resources etc. 
The current implementation is based on Linux procfs functionality and hence 
does not work on other platforms, specifically windows.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4204) Refactor ProcfsBasedProcessTree to make the resource collection object pluggable

Bikas Saha created MAPREDUCE-4204:
-

 Summary: Refactor ProcfsBasedProcessTree to make the resource 
collection object pluggable
 Key: MAPREDUCE-4204
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4204
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha


Making it a pluggable interface will allow replacing the procfs based 
implementation with ones for other platforms.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4202) TestYarnClientProtocolProvider is broken

[
https://issues.apache.org/jira/browse/MAPREDUCE-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263214#comment-13263214
]

Hadoop QA commented on MAPREDUCE-4202:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524770/MAPREDUCE-4202.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.conf.TestNoDefaultsJobConf

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2316//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2316//console

This message is automatically generated.

TestYarnClientProtocolProvider is broken

Key: MAPREDUCE-4202
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4202
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 0.24.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: MAPREDUCE-4202.patch

The test fails because a cluster is unexpectedly created with an empty conf.

[jira] [Commented] (MAPREDUCE-3871) Allow symlinking in LocalJobRunner DistributedCache

[
https://issues.apache.org/jira/browse/MAPREDUCE-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263217#comment-13263217
]

Hadoop QA commented on MAPREDUCE-3871:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524772/MAPREDUCE-3871.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens

org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService

org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry

org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2317//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2317//console

This message is automatically generated.

Allow symlinking in LocalJobRunner DistributedCache
---

Key: MAPREDUCE-3871
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3871
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: distributed-cache
Reporter: Tom White
Assignee: Tom White
Attachments: MAPREDUCE-3871.patch, MAPREDUCE-3871.patch

Currently the LocalJobRunner doesn't create symlinks for files in the
DistributedCache. It is safe to create symlinks if files of the same name
don't exist. LocalJobRunner should also delete the symlinks when the job has
completed.

[jira] [Commented] (MAPREDUCE-4088) Task stuck in JobLocalizer prevented other tasks on the same node from committing

2012-04-26 Thread Ravi Prakash (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263246#comment-13263246
 ] 

Ravi Prakash commented on MAPREDUCE-4088:
-

Does anyone have any suggestions on how to fix this?  I'm thinking of this: 
 * We could have a timeout in the wait for checkJobStatusAndWait. 
 * If we time out, we simply put the action back into the queue (hoping next 
time around it succeeds)
 * This might make the isIdle method more complicated :(


 Task stuck in JobLocalizer prevented other tasks on the same node from 
 committing
 -

 Key: MAPREDUCE-4088
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4088
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1
Affects Versions: 0.20.205.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Priority: Critical

 We saw that as a result of HADOOP-6963, one task was stuck in this
 Thread 23668: (state = IN_NATIVE)
  - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 
 (Compiled frame; information may be imprecise)
  - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 
 (Compiled frame)
  - java.io.File.exists() @bci=20, line=733 (Compiled frame)
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=3, line=446 
 (Compiled frame)
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Compiled frame)
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Compiled frame)
 
  TONS MORE OF THIS SAME LINE
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Compiled frame)
 .
 .
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Compiled frame)
  - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 
 (Interpreted frame)
 ne=451 (Interpreted frame)
  - 
 org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects(org.apache.hadoop.conf.Configuration,
  java.net.URI[], org.apache.hadoop.fs.Path[], long[], boolean[], boolean) 
 @bci=150, line=324 (Interpreted frame)
  - 
 org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCache(org.apache.hadoop.conf.Configuration)
  @bci=40, line=349 (Interpreted frame) 51, line=383 (Interpreted frame)
  - org.apache.hadoop.mapred.JobLocalizer.runSetup(java.lang.String, 
 java.lang.String, org.apache.hadoop.fs.Path, 
 org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=46, line=477 
 (Interpreted frame)
  - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=20, line=534 
 (Interpreted frame)
  - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=1, line=531 
 (Interpreted frame)
  - 
 java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
  java.security.AccessControlContext) @bci=0 (Interpreted frame)
  - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
 java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame)
  - 
 org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
  @bci=14, line=1082 (Interpreted frame)
  - org.apache.hadoop.mapred.JobLocalizer.main(java.lang.String[]) @bci=266, 
 line=530 (Interpreted frame)
 While all other tasks on the same node were stuck in 
 Thread 32141: (state = BLOCKED)
  - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
  - 
 org.apache.hadoop.mapred.Task.commit(org.apache.hadoop.mapred.TaskUmbilicalProtocol,
  org.apache.hadoop.mapred.Task$TaskReporter, 
 org.apache.hadoop.mapreduce.OutputCommitter) @bci=24, line=980 (Compiled 
 frame)
  - 
 org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol,
  org.apache.hadoop.mapred.Task$TaskReporter) @bci=146, line=871 (Interpreted 
 frame)
  - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf, 
 org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=470, line=423 
 (Interpreted frame)
  - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=255 (Interpreted 
 frame)
  - 
 java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction,
  java.security.AccessControlContext) @bci=0 (Interpreted frame)
  - javax.security.auth.Subject.doAs(javax.security.auth.Subject, 
 java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame)
  - 
 org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction)
  @bci=14, line=1082 (Interpreted frame)
  - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=738, line=249 
 (Interpreted frame)
 This should never happen. A stuck task should never prevent other tasks from 
 different jobs on the same node from committing.

--
This message is automatically generated by JIRA.
If you think it was sent

[jira] [Updated] (MAPREDUCE-4204) Refactor ProcfsBasedProcessTree to make the resource collection object pluggable


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bikas Saha updated MAPREDUCE-4204:
--

Attachment: MAPREDUCE-4204.patch

Refactor patch.
Replaces ProcfsBasedProcessTree with an abstract class 
ResourceCalculatorProcessTree. 

 Refactor ProcfsBasedProcessTree to make the resource collection object 
 pluggable
 

 Key: MAPREDUCE-4204
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4204
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: MAPREDUCE-4204.patch


 Making it a pluggable interface will allow replacing the procfs based 
 implementation with ones for other platforms.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3873) Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263322#comment-13263322
 ] 

xieguiming commented on MAPREDUCE-3873:
---

The modified function:
  public boolean isValidNode(String hostName) {
synchronized (hostsReader) {
  SetString hostsList = hostsReader.getHosts();
  SetString excludeList = hostsReader.getExcludedHosts();
  ｛color:red｝ 
  String ip = NetUtils.normalizeHostName(hostName);
   {color}
  return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList
  .contains(ip))
   !(excludeList.contains(hostName) || excludeList.contains(ip));
}
  }

and the function NetUtils.normalizeHostName(hostName) can not be mocked.
So, I think this patch do not need provide the unit testcase.

 Nodemanager is not getting decommisioned if the absolute ip is given in 
 exclude file.
 -

 Key: MAPREDUCE-3873
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3873
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Nishan Shetty
Priority: Minor
 Attachments: MAPREDUCE-3873.patch


 Configure absolute ip in yarn.resourcemanager.nodes.exclude-path and try to 
 decommission the node.
 It is not getting decommisioned.But if the hostname is given, decommissioning 
 is happening.
 I have also given the ip-host mapping of each machine in /etc/hosts.(i,e in 
 every machine the other machines ip-host mapping is specified).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-4052) Windows eclpise can not submit the job


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xieguiming reassigned MAPREDUCE-4052:
-

Assignee: xieguiming

 Windows eclpise can not submit the job
 --

 Key: MAPREDUCE-4052
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: job submission
Affects Versions: 0.23.1
 Environment: client on the Windows, the the cluster on the suse
Reporter: xieguiming
Assignee: xieguiming
 Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.patch


 when I use the eclipse on the windows to submit the job. and the 
 applicationmaster throw the exception:
 Exception in thread main java.lang.NoClassDefFoundError: 
 org/apache/hadoop/mapreduce/v2/app/MRAppMaster
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster
 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
 Could not find the main class: 
 org.apache.hadoop.mapreduce.v2.app.MRAppMaster.  Program will exit.
 The reasion is :
 class Apps addToEnvironment function, use the
 private static final String SYSTEM_PATH_SEPARATOR =
   System.getProperty(path.separator);
 and will result the MRApplicationMaster classpath use the ; separator.
 I suggest that nodemanger do the replace.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-3873) Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.