[jira] [Updated] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas updated MAPREDUCE-5176:
-

  Resolution: Fixed
Hadoop Flags: Incompatible change,Reviewed
  Status: Resolved  (was: Patch Available)

Committed to trunk, branch-2.

Thanks Carlo

 Preemptable annotations (to support preemption in MR)
 -

 Key: MAPREDUCE-5176
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Reporter: Carlo Curino
Assignee: Carlo Curino
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, 
 MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch


 Proposing a patch that introduces a new annotation @Preemptable that 
 represents to the framework property of user-supplied classes (e.g., Reducer, 
 OutputCommiter). The intended semantics is that a tagged class is safe to be 
 preempted between invocations. 
 (this is in spirit similar to the Output Contracts of [Nephele/PACT | 
 https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671198#comment-13671198
 ] 

Hudson commented on MAPREDUCE-5176:
---

Integrated in Hadoop-trunk-Commit #3820 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3820/])
MAPREDUCE-5176. Add annotation for tagging tasks as responsive to
preemption. Contributed by Carlo Curino, cdouglas (Revision 1488103)

 Result = SUCCESS
cdouglas : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488103
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation/Checkpointable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestPreemptableFileOutputCommitter.java


 Preemptable annotations (to support preemption in MR)
 -

 Key: MAPREDUCE-5176
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Reporter: Carlo Curino
Assignee: Carlo Curino
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, 
 MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch


 Proposing a patch that introduces a new annotation @Preemptable that 
 represents to the framework property of user-supplied classes (e.g., Reducer, 
 OutputCommiter). The intended semantics is that a tagged class is safe to be 
 preempted between invocations. 
 (this is in spirit similar to the Output Contracts of [Nephele/PACT | 
 https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Mayank Bansal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Bansal updated MAPREDUCE-5275:
-

Attachment: MAPREDUCE-5275-trunk-3.patch

Thanks [~vinodkv] for review and test.

Updating the latest patch.

Thanks,
Mayank

 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Mayank Bansal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Bansal updated MAPREDUCE-5275:
-

Status: Patch Available  (was: Open)

 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671208#comment-13671208
 ] 

Hadoop QA commented on MAPREDUCE-5275:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12585558/MAPREDUCE-5275-trunk-3.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:red}-1 findbugs{color}.  The patch appears to introduce 1 new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3707//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3707//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3707//console

This message is automatically generated.

 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Mayank Bansal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Bansal updated MAPREDUCE-5275:
-

Attachment: MAPREDUCE-5275-trunk-4.patch

Fixing the findbug warning.

Thanks,
Mayank

 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, 
 MAPREDUCE-5275-trunk-4.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Mayank Bansal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Bansal updated MAPREDUCE-5275:
-

Status: Open  (was: Patch Available)

 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, 
 MAPREDUCE-5275-trunk-4.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Mayank Bansal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Bansal updated MAPREDUCE-5275:
-

Status: Patch Available  (was: Open)

 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, 
 MAPREDUCE-5275-trunk-4.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671246#comment-13671246
 ] 

Hadoop QA commented on MAPREDUCE-5275:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12585561/MAPREDUCE-5275-trunk-4.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3708//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3708//console

This message is automatically generated.

 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, 
 MAPREDUCE-5275-trunk-4.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5036) Default shuffle handler port should not be 8080

[Tom White (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671311#comment-13671311
 ] 

Tom White commented on MAPREDUCE-5036:
--

+1 I'll commit this soon unless there are any objections to the new port number.

 Default shuffle handler port should not be 8080
 ---

 Key: MAPREDUCE-5036
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5036
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 2.0.3-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5036-13562.patch, MAPREDUCE-5036.patch


 The shuffle handler port (mapreduce.shuffle.port) defaults to 8080.  This is 
 a pretty common port for web services, and is likely to cause unnecessary 
 port conflicts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5261) TestRMContainerAllocator is exiting and failing the build

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671323#comment-13671323
 ] 

Hudson commented on MAPREDUCE-5261:
---

Integrated in Hadoop-Yarn-trunk #226 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/226/])
MAPREDUCE-5261. Reverting the patch as it is no longer needed. (Revision 
1488032)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488032
Files : 
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java


 TestRMContainerAllocator is exiting and failing the build
 -

 Key: MAPREDUCE-5261
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5261
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.5-beta
Reporter: Jason Lowe
Assignee: Omkar Vinit Joshi
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5261.20130521.1.patch, 
 MAPREDUCE-5261.20130521.2.patch, MAPREDUCE-5261.20130521.patch, 
 MAPREDUCE-5261.20130529.patch, MAPREDUCE-5261.20130529.patch, 
 MAPREDUCE-5261.patch


 Recent builds are failing because TestRMContainerAllocator is exiting rather 
 than succeeding or failing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671319#comment-13671319
 ] 

Hudson commented on MAPREDUCE-5176:
---

Integrated in Hadoop-Yarn-trunk #226 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/226/])
MAPREDUCE-5176. Add annotation for tagging tasks as responsive to
preemption. Contributed by Carlo Curino, cdouglas (Revision 1488103)

 Result = SUCCESS
cdouglas : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488103
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation/Checkpointable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestPreemptableFileOutputCommitter.java


 Preemptable annotations (to support preemption in MR)
 -

 Key: MAPREDUCE-5176
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Reporter: Carlo Curino
Assignee: Carlo Curino
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, 
 MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch


 Proposing a patch that introduces a new annotation @Preemptable that 
 represents to the framework property of user-supplied classes (e.g., Reducer, 
 OutputCommiter). The intended semantics is that a tagged class is safe to be 
 preempted between invocations. 
 (this is in spirit similar to the Output Contracts of [Nephele/PACT | 
 https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671318#comment-13671318
 ] 

Hudson commented on MAPREDUCE-5228:
---

Integrated in Hadoop-Yarn-trunk #226 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/226/])
MAPREDUCE-5228. Bring back FileInputFormat.Counter and 
FileOuputFormat.Counter for binary compatibility with 1.x mapred APIs. 
Contributed by Mayank Bansal. (Revision 1488060)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488060
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileOutputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java


 Enum Counter is removed from FileInputFormat and FileOutputFormat of both 
 mapred and mapreduce
 --

 Key: MAPREDUCE-5228
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5228-trunk-1.patch, 
 MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, 
 MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch, 
 MAPREDUCE-5228-trunk-6.patch


 The enum was used by findCounter(Enum key) to find a specific counter object. 
 Now it seems to be replaced by FileInputFormatCounter and 
 FileOutputFormatCounter. Now the enum seems to be only used internally, but 
 not sure whether it will be used externally when users extend FileFormat.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5282) Update MR App to use immutable ApplicationID after YARN-716

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671326#comment-13671326
 ] 

Hudson commented on MAPREDUCE-5282:
---

Integrated in Hadoop-Yarn-trunk #226 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/226/])
YARN-716. Making ApplicationID immutable. Contributed by Siddharth Seth.
MAPREDUCE-5282. Updating MR App to use immutable ApplicationID after YARN-716. 
Contributed by Siddharth Seth. (Revision 1487994)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1487994
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAppController.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestShufflePlugin.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationIdPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestEventFlow.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerReboot.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
* 

[jira] [Commented] (MAPREDUCE-5286) startContainer call should use the ContainerToken instead of Container [YARN-684]

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671329#comment-13671329
 ] 

Hudson commented on MAPREDUCE-5286:
---

Integrated in Hadoop-Yarn-trunk #226 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/226/])
MAPREDUCE-5286. Change MapReduce to use ContainerTokenIdentifier instead of 
the entire Container in the startContainer call - YARN-684. Contributed by 
Vinod Kumar Vavilapalli. (Revision 1488087)

 Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488087
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java


 startContainer call should use the ContainerToken instead of Container 
 [YARN-684]
 -

 Key: MAPREDUCE-5286
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5286
 Project: Hadoop Map/Reduce
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Vinod Kumar Vavilapalli
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5286.txt


 MapReduce counterpart of YARN-684.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-5287) Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks

[nicu marasoiu (JIRA)]

nicu marasoiu created MAPREDUCE-5287:


 Summary: Create a generic InputFormat wrapping any other 
InputFormat, to control the number of map tasks
 Key: MAPREDUCE-5287
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5287
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, performance
Reporter: nicu marasoiu


I wrote a generic InputFormat that wraps any other InputFormat, and creates 
CompositeInputSplits to reduce the number of map tasks in a controllable manner 
while preserving data locality. A correspondent CompositeRecordReader is 
written to iterate through underlying RecordReaders as created by the 
underlying InputFormat for each underlying raw split.

An application to this is to group TableSplits when the raw splits are coming 
from multiple regions and are filtered with key ranges. We use this to 
shard/distribute a time based incremental access to an hbase table.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671426#comment-13671426
 ] 

Hudson commented on MAPREDUCE-5176:
---

Integrated in Hadoop-Hdfs-trunk #1416 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1416/])
MAPREDUCE-5176. Add annotation for tagging tasks as responsive to
preemption. Contributed by Carlo Curino, cdouglas (Revision 1488103)

 Result = FAILURE
cdouglas : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488103
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation/Checkpointable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestPreemptableFileOutputCommitter.java


 Preemptable annotations (to support preemption in MR)
 -

 Key: MAPREDUCE-5176
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Reporter: Carlo Curino
Assignee: Carlo Curino
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, 
 MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch


 Proposing a patch that introduces a new annotation @Preemptable that 
 represents to the framework property of user-supplied classes (e.g., Reducer, 
 OutputCommiter). The intended semantics is that a tagged class is safe to be 
 preempted between invocations. 
 (this is in spirit similar to the Output Contracts of [Nephele/PACT | 
 https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671425#comment-13671425
 ] 

Hudson commented on MAPREDUCE-5228:
---

Integrated in Hadoop-Hdfs-trunk #1416 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1416/])
MAPREDUCE-5228. Bring back FileInputFormat.Counter and 
FileOuputFormat.Counter for binary compatibility with 1.x mapred APIs. 
Contributed by Mayank Bansal. (Revision 1488060)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488060
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileOutputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java


 Enum Counter is removed from FileInputFormat and FileOutputFormat of both 
 mapred and mapreduce
 --

 Key: MAPREDUCE-5228
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5228-trunk-1.patch, 
 MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, 
 MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch, 
 MAPREDUCE-5228-trunk-6.patch


 The enum was used by findCounter(Enum key) to find a specific counter object. 
 Now it seems to be replaced by FileInputFormatCounter and 
 FileOutputFormatCounter. Now the enum seems to be only used internally, but 
 not sure whether it will be used externally when users extend FileFormat.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5261) TestRMContainerAllocator is exiting and failing the build

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671431#comment-13671431
 ] 

Hudson commented on MAPREDUCE-5261:
---

Integrated in Hadoop-Hdfs-trunk #1416 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1416/])
MAPREDUCE-5261. Reverting the patch as it is no longer needed. (Revision 
1488032)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488032
Files : 
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java


 TestRMContainerAllocator is exiting and failing the build
 -

 Key: MAPREDUCE-5261
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5261
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.5-beta
Reporter: Jason Lowe
Assignee: Omkar Vinit Joshi
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5261.20130521.1.patch, 
 MAPREDUCE-5261.20130521.2.patch, MAPREDUCE-5261.20130521.patch, 
 MAPREDUCE-5261.20130529.patch, MAPREDUCE-5261.20130529.patch, 
 MAPREDUCE-5261.patch


 Recent builds are failing because TestRMContainerAllocator is exiting rather 
 than succeeding or failing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5282) Update MR App to use immutable ApplicationID after YARN-716

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671434#comment-13671434
 ] 

Hudson commented on MAPREDUCE-5282:
---

Integrated in Hadoop-Hdfs-trunk #1416 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1416/])
YARN-716. Making ApplicationID immutable. Contributed by Siddharth Seth.
MAPREDUCE-5282. Updating MR App to use immutable ApplicationID after YARN-716. 
Contributed by Siddharth Seth. (Revision 1487994)

 Result = FAILURE
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1487994
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAppController.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestShufflePlugin.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationIdPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestEventFlow.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerReboot.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
* 

[jira] [Commented] (MAPREDUCE-5286) startContainer call should use the ContainerToken instead of Container [YARN-684]

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671437#comment-13671437
 ] 

Hudson commented on MAPREDUCE-5286:
---

Integrated in Hadoop-Hdfs-trunk #1416 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1416/])
MAPREDUCE-5286. Change MapReduce to use ContainerTokenIdentifier instead of 
the entire Container in the startContainer call - YARN-684. Contributed by 
Vinod Kumar Vavilapalli. (Revision 1488087)

 Result = FAILURE
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488087
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java


 startContainer call should use the ContainerToken instead of Container 
 [YARN-684]
 -

 Key: MAPREDUCE-5286
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5286
 Project: Hadoop Map/Reduce
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Vinod Kumar Vavilapalli
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5286.txt


 MapReduce counterpart of YARN-684.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671490#comment-13671490
 ] 

Hudson commented on MAPREDUCE-5176:
---

Integrated in Hadoop-Mapreduce-trunk #1442 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1442/])
MAPREDUCE-5176. Add annotation for tagging tasks as responsive to
preemption. Contributed by Carlo Curino, cdouglas (Revision 1488103)

 Result = SUCCESS
cdouglas : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488103
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialOutputCommitter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation/Checkpointable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestPreemptableFileOutputCommitter.java


 Preemptable annotations (to support preemption in MR)
 -

 Key: MAPREDUCE-5176
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Reporter: Carlo Curino
Assignee: Carlo Curino
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, 
 MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch


 Proposing a patch that introduces a new annotation @Preemptable that 
 represents to the framework property of user-supplied classes (e.g., Reducer, 
 OutputCommiter). The intended semantics is that a tagged class is safe to be 
 preempted between invocations. 
 (this is in spirit similar to the Output Contracts of [Nephele/PACT | 
 https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671489#comment-13671489
 ] 

Hudson commented on MAPREDUCE-5228:
---

Integrated in Hadoop-Mapreduce-trunk #1442 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1442/])
MAPREDUCE-5228. Bring back FileInputFormat.Counter and 
FileOuputFormat.Counter for binary compatibility with 1.x mapred APIs. 
Contributed by Mayank Bansal. (Revision 1488060)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488060
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileOutputFormat_Counter.properties
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java


 Enum Counter is removed from FileInputFormat and FileOutputFormat of both 
 mapred and mapreduce
 --

 Key: MAPREDUCE-5228
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5228-trunk-1.patch, 
 MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, 
 MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch, 
 MAPREDUCE-5228-trunk-6.patch


 The enum was used by findCounter(Enum key) to find a specific counter object. 
 Now it seems to be replaced by FileInputFormatCounter and 
 FileOutputFormatCounter. Now the enum seems to be only used internally, but 
 not sure whether it will be used externally when users extend FileFormat.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5261) TestRMContainerAllocator is exiting and failing the build

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671494#comment-13671494
 ] 

Hudson commented on MAPREDUCE-5261:
---

Integrated in Hadoop-Mapreduce-trunk #1442 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1442/])
MAPREDUCE-5261. Reverting the patch as it is no longer needed. (Revision 
1488032)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488032
Files : 
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java


 TestRMContainerAllocator is exiting and failing the build
 -

 Key: MAPREDUCE-5261
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5261
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.5-beta
Reporter: Jason Lowe
Assignee: Omkar Vinit Joshi
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5261.20130521.1.patch, 
 MAPREDUCE-5261.20130521.2.patch, MAPREDUCE-5261.20130521.patch, 
 MAPREDUCE-5261.20130529.patch, MAPREDUCE-5261.20130529.patch, 
 MAPREDUCE-5261.patch


 Recent builds are failing because TestRMContainerAllocator is exiting rather 
 than succeeding or failing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5282) Update MR App to use immutable ApplicationID after YARN-716

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671497#comment-13671497
 ] 

Hudson commented on MAPREDUCE-5282:
---

Integrated in Hadoop-Mapreduce-trunk #1442 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1442/])
YARN-716. Making ApplicationID immutable. Contributed by Siddharth Seth.
MAPREDUCE-5282. Updating MR App to use immutable ApplicationID after YARN-716. 
Contributed by Siddharth Seth. (Revision 1487994)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1487994
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAppController.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestShufflePlugin.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationIdPBImpl.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestEventFlow.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerReboot.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java
* 
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
* 

[jira] [Commented] (MAPREDUCE-5286) startContainer call should use the ContainerToken instead of Container [YARN-684]

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671500#comment-13671500
 ] 

Hudson commented on MAPREDUCE-5286:
---

Integrated in Hadoop-Mapreduce-trunk #1442 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1442/])
MAPREDUCE-5286. Change MapReduce to use ContainerTokenIdentifier instead of 
the entire Container in the startContainer call - YARN-684. Contributed by 
Vinod Kumar Vavilapalli. (Revision 1488087)

 Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488087
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java


 startContainer call should use the ContainerToken instead of Container 
 [YARN-684]
 -

 Key: MAPREDUCE-5286
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5286
 Project: Hadoop Map/Reduce
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Vinod Kumar Vavilapalli
 Fix For: 2.0.5-beta

 Attachments: MAPREDUCE-5286.txt


 MapReduce counterpart of YARN-684.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671535#comment-13671535
 ] 

Jason Lowe commented on MAPREDUCE-5268:
---

Thanks for updating so quickly, Karthik.  Sorry, I should have been more clear 
on my nit with the templates.  I was surprised to see such a generic type, both 
in name and template use, for a specific, private use-case.  If the name is 
something so generic as ConcurrentSkipListMapWithSize then I think it is 
appropriate to use templates with that name since the name implies it is 
generic.  If we think the templates are overkill then I think it's better to 
name it more specifically for its purpose, e.g.: JobHistoryInfoMap.

I do like the AtomicInteger change.  And as far as the size inconsistency 
concern, that's already an issue because of the very nature of concurrency 
whether we synchronize the methods or not.  As soon as we get/compute the size, 
it could be wrong as someone else comes in and changes the map immediately 
afterwards. In this case the size is only being used to loosely cap the size of 
the job cache, and it's not critical if we're a few jobs over or a few jobs 
under that limit.

So I think we're really close.  I'd like to see either the 
ConcurrentSkipListMapWithSize name changed to be more specific to its use case 
or keep the name and have the templates restored.  I don't care too much either 
way.

 Improve history server startup performance
 --

 Key: MAPREDUCE-5268
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Karthik Kambatla
 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, 
 mr-5268-prelim.patch


 The history server can easily take many minutes to startup when there are a 
 significant number of jobs to scan in the done directory.  However the 
 scanning of files is not the bottleneck, rather it's the heavy use of 
 ConcurrentSkipListMap.size in HistoryFileManager.  
 ConcurrentSkipListMap.size is a very expensive operation, especially on maps 
 with many entries, as it has to scan every entry to compute the size.  We 
 should avoid calling this method or at least minimize its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5268) Improve history server startup performance

[Karthik Kambatla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-5268:


Attachment: mr-5268.patch

Updated the helper class name to JobIdHistoryFileInfoMap

 Improve history server startup performance
 --

 Key: MAPREDUCE-5268
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Karthik Kambatla
 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, 
 mr-5268.patch, mr-5268-prelim.patch


 The history server can easily take many minutes to startup when there are a 
 significant number of jobs to scan in the done directory.  However the 
 scanning of files is not the bottleneck, rather it's the heavy use of 
 ConcurrentSkipListMap.size in HistoryFileManager.  
 ConcurrentSkipListMap.size is a very expensive operation, especially on maps 
 with many entries, as it has to scan every entry to compute the size.  We 
 should avoid calling this method or at least minimize its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671571#comment-13671571
 ] 

Hadoop QA commented on MAPREDUCE-5268:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585626/mr-5268.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3709//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3709//console

This message is automatically generated.

 Improve history server startup performance
 --

 Key: MAPREDUCE-5268
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Karthik Kambatla
 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, 
 mr-5268.patch, mr-5268-prelim.patch


 The history server can easily take many minutes to startup when there are a 
 significant number of jobs to scan in the done directory.  However the 
 scanning of files is not the bottleneck, rather it's the heavy use of 
 ConcurrentSkipListMap.size in HistoryFileManager.  
 ConcurrentSkipListMap.size is a very expensive operation, especially on maps 
 with many entries, as it has to scan every entry to compute the size.  We 
 should avoid calling this method or at least minimize its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671582#comment-13671582
 ] 

Jason Lowe commented on MAPREDUCE-5268:
---

Wow, that was fast!  Unfortunately ConcurrentSkipListMapWithSize is still 
showing up in filenames.

Also taking a closer look at the unit tests, FakeJobId should be removed and 
replaced with MRBuilderUtils.newJobId.  There's a form of that method where we 
can simply pass numbers for the cluster timestamp, app ID, and job ID directly.

 Improve history server startup performance
 --

 Key: MAPREDUCE-5268
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Karthik Kambatla
 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, 
 mr-5268.patch, mr-5268-prelim.patch


 The history server can easily take many minutes to startup when there are a 
 significant number of jobs to scan in the done directory.  However the 
 scanning of files is not the bottleneck, rather it's the heavy use of 
 ConcurrentSkipListMap.size in HistoryFileManager.  
 ConcurrentSkipListMap.size is a very expensive operation, especially on maps 
 with many entries, as it has to scan every entry to compute the size.  We 
 should avoid calling this method or at least minimize its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5199) AppTokens file can/should be removed

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated MAPREDUCE-5199:
---

Attachment: MAPREDUCE-5199.patch

bq. JobImpl.fsTokens should be renamed too.
Done

bq. Jobtoken itself could be set in jobCredentials and passed to TaskImpl etc. 
Perhaps another ticket.
Yes, will defer since it's out of scope for removing appTokens file.

bq. initAndStartAppMaster Adding the credentials to conf is necessary for the 
sake of MAPREDUCE-5240. Please remove the comment  // shouldn't be necessary
Removed.

bq. And it is better to add to conf before init() happens. We never know if the 
original conf is cached as part of init().
Done.  Although the first thing init() does is call the method to setup the 
credentials, I changed it.

bq. appMasterUgi is useless. Perhaps remove it in a separate ticket.
I too thought it was useless, but it's not.  That's how the AM constructs a UGI 
as the submitting user in insecure mode.  Otherwise the UGI is the NM user.

The issue is the launch context sets $USER which in turn is used by the AM to 
construct the unnecessary UGI.  However if $HADOOP_USER_NAME=$USER is also 
defined in the launch context, the UGI will create the login UGI as the 
submitter.

bq. TestMRAppMaster: Wherever you say appToken shouldn't be present, you should 
explicitly validate it. Seems like that comment is invalid in some cases.
It does validate by checking that there's 1 token, and that it's the expected 
non-app token.

bq.  MRJobConf.APPLICATION_TOKENS_FILE should just be removed along with the 
java comments. This was supposed to be private anyways.
Done.  I considered this too, but was worried if other projects might be using 
it.

bq. downloadTokensAndSetupUGI - setupJobCredentialsAndUGI like Sid proposed.
Done.

 AppTokens file can/should be removed
 

 Key: MAPREDUCE-5199
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: security
Affects Versions: 3.0.0, 2.0.5-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: MAPREDUCE-5199.patch, MAPREDUCE-5199.patch


 All the required tokens are propagated to AMs and containers via 
 startContainer(), no need for explicitly creating the app-token file that we 
 have today..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5199) AppTokens file can/should be removed

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated MAPREDUCE-5199:
---

Status: Patch Available  (was: Open)

 AppTokens file can/should be removed
 

 Key: MAPREDUCE-5199
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: security
Affects Versions: 3.0.0, 2.0.5-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: MAPREDUCE-5199.patch, MAPREDUCE-5199.patch


 All the required tokens are propagated to AMs and containers via 
 startContainer(), no need for explicitly creating the app-token file that we 
 have today..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5199) AppTokens file can/should be removed

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671602#comment-13671602
 ] 

Hadoop QA commented on MAPREDUCE-5199:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585632/MAPREDUCE-5199.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:red}-1 javac{color:red}.  The patch appears to cause the build to 
fail.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3710//console

This message is automatically generated.

 AppTokens file can/should be removed
 

 Key: MAPREDUCE-5199
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: security
Affects Versions: 3.0.0, 2.0.5-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: MAPREDUCE-5199.patch, MAPREDUCE-5199.patch


 All the required tokens are propagated to AMs and containers via 
 startContainer(), no need for explicitly creating the app-token file that we 
 have today..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4019) -list-attempt-ids is not working

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671614#comment-13671614
 ] 

Jason Lowe commented on MAPREDUCE-4019:
---

Thanks for picking up the torch on this, Ashwin.  Couple of comments:

* Whitespace was removed before/after a number of methods which would be nice 
to have back to match the style of the rest of the code
* getCluster should be named something like createCluster to indicate it is 
creating a new object each time it's called
* getTaskTypes() can simply be StringUtils.join(taskTypes,  )

 -list-attempt-ids  is not working
 -

 Key: MAPREDUCE-4019
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Priority: Minor
 Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019.patch, 
 MAPREDUCE-4019.patch, MAPREDUCE-4019.txt


 while executing {noformat}bin/mapred  job -list-attempt-ids job_id map 
 running{noformat}, we are getting IllegalArgumentexception.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5268) Improve history server startup performance

[Karthik Kambatla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-5268:


Attachment: mr-5268.patch

Thanks for the patient reviews, Jason.

My bad forgot to rename the corresponding test - fixed that now. Also, the 
tests now use MRBuilderUtils instead of FakeJob.

Reviewed the patch carefully, hopefully this takes care of everything.

 Improve history server startup performance
 --

 Key: MAPREDUCE-5268
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Karthik Kambatla
 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, 
 mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch


 The history server can easily take many minutes to startup when there are a 
 significant number of jobs to scan in the done directory.  However the 
 scanning of files is not the bottleneck, rather it's the heavy use of 
 ConcurrentSkipListMap.size in HistoryFileManager.  
 ConcurrentSkipListMap.size is a very expensive operation, especially on maps 
 with many entries, as it has to scan every entry to compute the size.  We 
 should avoid calling this method or at least minimize its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671705#comment-13671705
 ] 

Hadoop QA commented on MAPREDUCE-5268:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585646/mr-5268.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3711//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3711//console

This message is automatically generated.

 Improve history server startup performance
 --

 Key: MAPREDUCE-5268
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Karthik Kambatla
 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, 
 mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch


 The history server can easily take many minutes to startup when there are a 
 significant number of jobs to scan in the done directory.  However the 
 scanning of files is not the bottleneck, rather it's the heavy use of 
 ConcurrentSkipListMap.size in HistoryFileManager.  
 ConcurrentSkipListMap.size is a very expensive operation, especially on maps 
 with many entries, as it has to scan every entry to compute the size.  We 
 should avoid calling this method or at least minimize its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5199) AppTokens file can/should be removed

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated MAPREDUCE-5199:
---

Attachment: MAPREDUCE-5199.patch

Uploaded wrong patch.

 AppTokens file can/should be removed
 

 Key: MAPREDUCE-5199
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: security
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, 
 MAPREDUCE-5199.patch


 All the required tokens are propagated to AMs and containers via 
 startContainer(), no need for explicitly creating the app-token file that we 
 have today..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5281) Mapreduce API: Counter changes from non-abstract class to interface

[Zhijie Shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671722#comment-13671722
 ] 

Zhijie Shen commented on MAPREDUCE-5281:


As Counter becomes interface, there's no way to add the implemented methods. 
However, the four aforementioned methods must exist in the interface's 
implementations. boolean equals(Object) and int hashCode() comes from Object, 
while void readFields(DataInput) and void write(DataOutput) will be implemented 
as they are part of the interface (Counter extends Writable).

 Mapreduce API: Counter changes from non-abstract class to interface
 ---

 Key: MAPREDUCE-5281
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5281
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen

 Therefore, significant changes in Counter:
 1. Two Constructors are removed;
 2. Following methods are removed:
 * boolean equals(Object)
 * int hashCode()
 * void readFields(DataInput)
 * void write(DataOutput)
 Fix of this issue may break 0.23.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5199) AppTokens file can/should be removed

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671756#comment-13671756
 ] 

Hadoop QA commented on MAPREDUCE-5199:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12585649/MAPREDUCE-5199.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 5 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

  org.apache.hadoop.mapreduce.v2.app.TestRecovery
  org.apache.hadoop.mapreduce.v2.app.TestMRApp

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3712//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3712//console

This message is automatically generated.

 AppTokens file can/should be removed
 

 Key: MAPREDUCE-5199
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: security
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Daryn Sharp
Priority: Blocker
 Attachments: MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, 
 MAPREDUCE-5199.patch


 All the required tokens are propagated to AMs and containers via 
 startContainer(), no need for explicitly creating the app-token file that we 
 have today..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671774#comment-13671774
 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-5275:


+1, looks good now. Checking this in..

 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, 
 MAPREDUCE-5275-trunk-4.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-5275:
---

   Resolution: Fixed
Fix Version/s: 2.1.0-beta
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed this to trunk and branch-2. Thanks Mayank!

Tx for the early review Zhijie!

 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Fix For: 2.1.0-beta

 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, 
 MAPREDUCE-5275-trunk-4.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671782#comment-13671782
 ] 

Hudson commented on MAPREDUCE-5275:
---

Integrated in Hadoop-trunk-Commit #3822 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3822/])
MAPREDUCE-5275. Bring back a couple of APIs in 
mapreduce.security.TokenCache for binary compatibility with 1.x mapreduce APIs. 
Contributed by Mayank Bansal. (Revision 1488369)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488369
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/TokenCache.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/security/TestTokenCache.java


 Mapreduce API: TokenCache incompatibility issues with MR1
 -

 Key: MAPREDUCE-5275
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Fix For: 2.1.0-beta

 Attachments: MAPREDUCE-5275-trunk-1.patch, 
 MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch, 
 MAPREDUCE-5275-trunk-4.patch


 There're following incompatibility issues:
 * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is 
 removed
 * Credentials loadTokens(String, Configuration) changes to Credentials 
 loadTokens(String, JobConf)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671798#comment-13671798
 ] 

Jason Lowe commented on MAPREDUCE-5268:
---

+1 for the latest patch.

Thanks so much for the rapid turnarounds on the patches and your patience, 
Karthik!

 Improve history server startup performance
 --

 Key: MAPREDUCE-5268
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Karthik Kambatla
 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, 
 mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch


 The history server can easily take many minutes to startup when there are a 
 significant number of jobs to scan in the done directory.  However the 
 scanning of files is not the bottleneck, rather it's the heavy use of 
 ConcurrentSkipListMap.size in HistoryFileManager.  
 ConcurrentSkipListMap.size is a very expensive operation, especially on maps 
 with many entries, as it has to scan every entry to compute the size.  We 
 should avoid calling this method or at least minimize its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-4404) Adding Test case for TestMRJobClient to verify the user name

[Mayank Bansal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Bansal resolved MAPREDUCE-4404.
--

Resolution: Fixed

 Adding Test case for TestMRJobClient to verify the user name
 

 Key: MAPREDUCE-4404
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4404
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Priority: Minor
 Fix For: 0.22.1

 Attachments: MAPREDUCE-4404-22.patch


 Adding Test case for TestMRJobClient to verify the user name

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5192) Separate TCE resolution from fetch

[Arun C Murthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy updated MAPREDUCE-5192:
-

Assignee: Chris Douglas

 Separate TCE resolution from fetch
 --

 Key: MAPREDUCE-5192
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5192
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: task
Reporter: Chris Douglas
Assignee: Chris Douglas
Priority: Minor
 Fix For: 2.1.0-beta

 Attachments: M5192-1.patch, M5192-2.patch, M5192.patch


 The {{EventFetcher}} thread grounds task completion events as URIs before 
 passing them to the {{ShuffleScheduler}}. If the former deferred this to the 
 scheduler, one could interpret the TCE metadata differently

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext

[Sandy Ryza (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sandy Ryza updated MAPREDUCE-5283:
--

Attachment: MAPREDUCE-5283-1.patch

 Over 10 different tests have near identical implementations of AppContext
 -

 Key: MAPREDUCE-5283
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: applicationmaster, test
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283.patch


 I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go 
 into nearly every test file for MR web services to make sure their 
 TestAppContext implements it.  I propose having a common implementation of 
 AppContext that all these tests can use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671851#comment-13671851
 ] 

Hadoop QA commented on MAPREDUCE-5283:
--

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12585671/MAPREDUCE-5283-1.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 15 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3713//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3713//console

This message is automatically generated.

 Over 10 different tests have near identical implementations of AppContext
 -

 Key: MAPREDUCE-5283
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: applicationmaster, test
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283.patch


 I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go 
 into nearly every test file for MR web services to make sure their 
 TestAppContext implements it.  I propose having a common implementation of 
 AppContext that all these tests can use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671884#comment-13671884
 ] 

Jason Lowe commented on MAPREDUCE-5268:
---

Will commit this after the 2.0.5 / 2.1.0 dust has cleared which should be done 
this weekend.

 Improve history server startup performance
 --

 Key: MAPREDUCE-5268
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Karthik Kambatla
 Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, 
 mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch


 The history server can easily take many minutes to startup when there are a 
 significant number of jobs to scan in the done directory.  However the 
 scanning of files is not the bottleneck, rather it's the heavy use of 
 ConcurrentSkipListMap.size in HistoryFileManager.  
 ConcurrentSkipListMap.size is a very expensive operation, especially on maps 
 with many entries, as it has to scan every entry to compute the size.  We 
 should avoid calling this method or at least minimize its use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5150) Backport 2009 terasort (MAPREDUCE-639) to branch-1

[Gera Shegalov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gera Shegalov updated MAPREDUCE-5150:
-

Affects Version/s: 1.2.0

 Backport 2009 terasort (MAPREDUCE-639) to branch-1
 --

 Key: MAPREDUCE-5150
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5150
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: examples
Affects Versions: 1.0.4, 1.2.0
Reporter: Gera Shegalov
Priority: Minor
 Attachments: MAPREDUCE-5150-branch-1.patch


 Users evaluate performance of Hadoop clusters using different benchmarks such 
 as TeraSort. However, terasort version in branch-1 is outdated. It works on 
 teragen dataset that cannot exceed 4 billion unique keys and it does not have 
 the fast non-sampling partitioner SimplePartitioner either.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5150) Backport 2009 terasort (MAPREDUCE-639) to branch-1

[Gera Shegalov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gera Shegalov updated MAPREDUCE-5150:
-

Affects Version/s: (was: 1.0.4)

 Backport 2009 terasort (MAPREDUCE-639) to branch-1
 --

 Key: MAPREDUCE-5150
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5150
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: examples
Affects Versions: 1.2.0
Reporter: Gera Shegalov
Priority: Minor
 Attachments: MAPREDUCE-5150-branch-1.patch


 Users evaluate performance of Hadoop clusters using different benchmarks such 
 as TeraSort. However, terasort version in branch-1 is outdated. It works on 
 teragen dataset that cannot exceed 4 billion unique keys and it does not have 
 the fast non-sampling partitioner SimplePartitioner either.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5150) Backport 2009 terasort (MAPREDUCE-639) to branch-1

[Gera Shegalov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gera Shegalov updated MAPREDUCE-5150:
-

Status: Patch Available  (was: Open)

 Backport 2009 terasort (MAPREDUCE-639) to branch-1
 --

 Key: MAPREDUCE-5150
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5150
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: examples
Affects Versions: 1.2.0
Reporter: Gera Shegalov
Priority: Minor
 Attachments: MAPREDUCE-5150-branch-1.patch


 Users evaluate performance of Hadoop clusters using different benchmarks such 
 as TeraSort. However, terasort version in branch-1 is outdated. It works on 
 teragen dataset that cannot exceed 4 billion unique keys and it does not have 
 the fast non-sampling partitioner SimplePartitioner either.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671890#comment-13671890
 ] 

Jason Lowe commented on MAPREDUCE-5283:
---

+1, lgtm.  Waiting for the dust to settle this weekend on 2.0.5 / 2.1.0 and to 
give others a chance to comment.

 Over 10 different tests have near identical implementations of AppContext
 -

 Key: MAPREDUCE-5283
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: applicationmaster, test
Affects Versions: 2.0.4-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
 Attachments: MAPREDUCE-5283-1.patch, MAPREDUCE-5283.patch


 I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go 
 into nearly every test file for MR web services to make sure their 
 TestAppContext implements it.  I propose having a common implementation of 
 AppContext that all these tests can use.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5150) Backport 2009 terasort (MAPREDUCE-639) to branch-1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671892#comment-13671892
 ] 

Hadoop QA commented on MAPREDUCE-5150:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12578622/MAPREDUCE-5150-branch-1.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3714//console

This message is automatically generated.

 Backport 2009 terasort (MAPREDUCE-639) to branch-1
 --

 Key: MAPREDUCE-5150
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5150
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: examples
Affects Versions: 1.2.0
Reporter: Gera Shegalov
Priority: Minor
 Attachments: MAPREDUCE-5150-branch-1.patch


 Users evaluate performance of Hadoop clusters using different benchmarks such 
 as TeraSort. However, terasort version in branch-1 is outdated. It works on 
 teragen dataset that cannot exceed 4 billion unique keys and it does not have 
 the fast non-sampling partitioner SimplePartitioner either.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5231) Constructor of DBInputFormat.DBRecordReader in mapred is changed

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671901#comment-13671901
 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-5231:


This DB* code is a bit of a mind-bender w.r.t mapred and mapreduce impls.

Looks good, +1. Checking this in.

 Constructor of DBInputFormat.DBRecordReader in mapred is changed
 

 Key: MAPREDUCE-5231
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5231
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Attachments: MAPREDUCE-5231.1.patch


 The constructor of DBInputFormat.DBRecordReader in mapred is changed from MR1 
 to RM2. Though MAPREDUCE-716 tried to deal with the API difference. However, 
 if DBInputFormat.DBRecordReader is extended, the incompatibility around the 
 constructor is still there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5231) Constructor of DBInputFormat.DBRecordReader in mapred is changed

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-5231:
---

   Resolution: Fixed
Fix Version/s: 2.1.0-beta
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed this to trunk and branch-2. Thanks Zhijie!

 Constructor of DBInputFormat.DBRecordReader in mapred is changed
 

 Key: MAPREDUCE-5231
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5231
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.1.0-beta

 Attachments: MAPREDUCE-5231.1.patch


 The constructor of DBInputFormat.DBRecordReader in mapred is changed from MR1 
 to RM2. Though MAPREDUCE-716 tried to deal with the API difference. However, 
 if DBInputFormat.DBRecordReader is extended, the incompatibility around the 
 constructor is still there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5231) Constructor of DBInputFormat.DBRecordReader in mapred is changed

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671915#comment-13671915
 ] 

Hudson commented on MAPREDUCE-5231:
---

Integrated in Hadoop-trunk-Commit #3823 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3823/])
MAPREDUCE-5231. Bring back a constructor in mapred's 
DBInputFormat.DBRecordReader for binary compatibility with 1.x mapred APIs. 
Contributed by Zhijie Shen. (Revision 1488436)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488436
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/db/DBInputFormat.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java


 Constructor of DBInputFormat.DBRecordReader in mapred is changed
 

 Key: MAPREDUCE-5231
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5231
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
 Fix For: 2.1.0-beta

 Attachments: MAPREDUCE-5231.1.patch


 The constructor of DBInputFormat.DBRecordReader in mapred is changed from MR1 
 to RM2. Though MAPREDUCE-716 tried to deal with the API difference. However, 
 if DBInputFormat.DBRecordReader is extended, the incompatibility around the 
 constructor is still there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-5285) Update MR App to use immutable ApplicationAttemptID, ContainerID, NodeID after YARN-735

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth resolved MAPREDUCE-5285.
---

   Resolution: Fixed
Fix Version/s: 2.1.0-beta

Committed as part of YARN-735.

 Update MR App to use immutable ApplicationAttemptID, ContainerID, NodeID 
 after YARN-735
 ---

 Key: MAPREDUCE-5285
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5285
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Jian He
 Fix For: 2.1.0-beta




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5273) Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671925#comment-13671925
 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-5273:


It's so trivial it's funny. These variables aren't even used otherwise.

+1. Checking this in.


 Protected variables are removed from CombineFileRecordReader in both mapred 
 and mapreduce
 -

 Key: MAPREDUCE-5273
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5273
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5273-trunk-1.patch


 Two protected variables are removed from CombineFileRecordReader in both 
 mapred and mapreduce:
 * FileSystem fs
 * ClassRecordReaderK, V rrClass

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5273) Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-5273:
---

   Resolution: Fixed
Fix Version/s: 2.1.0-beta
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed this to trunk and branch-2. Thanks Mayank!

 Protected variables are removed from CombineFileRecordReader in both mapred 
 and mapreduce
 -

 Key: MAPREDUCE-5273
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5273
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Fix For: 2.1.0-beta

 Attachments: MAPREDUCE-5273-trunk-1.patch


 Two protected variables are removed from CombineFileRecordReader in both 
 mapred and mapreduce:
 * FileSystem fs
 * ClassRecordReaderK, V rrClass

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5273) Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671936#comment-13671936
 ] 

Hudson commented on MAPREDUCE-5273:
---

Integrated in Hadoop-trunk-Commit #3825 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3825/])
MAPREDUCE-5273. Bring back a couple of protected variables in mapred and 
mapreduce CombineFileRecordReader for binary compatibility with 1.x APIs. 
Contributed by Mayank Bansal. (Revision 1488442)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488442
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileRecordReader.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileRecordReader.java


 Protected variables are removed from CombineFileRecordReader in both mapred 
 and mapreduce
 -

 Key: MAPREDUCE-5273
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5273
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Fix For: 2.1.0-beta

 Attachments: MAPREDUCE-5273-trunk-1.patch


 Two protected variables are removed from CombineFileRecordReader in both 
 mapred and mapreduce:
 * FileSystem fs
 * ClassRecordReaderK, V rrClass

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5280) Mapreduce API: ClusterMetrics incompatibility issues with MR1

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671956#comment-13671956
 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-5280:


+1, looks good. Checking this in.

 Mapreduce API: ClusterMetrics incompatibility issues with MR1
 -

 Key: MAPREDUCE-5280
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5280
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-5280-trunk-1.patch


 1. Constructor has one fewer parameters: numGraylistedTrackers
 2. getGrayListedTaskTrackerCount() is removed

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-5280) Mapreduce API: ClusterMetrics incompatibility issues with MR1

[Vinod Kumar Vavilapalli (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-5280:
---

   Resolution: Fixed
Fix Version/s: 2.1.0-beta
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed this to trunk and branch-2. Thanks Mayank!

 Mapreduce API: ClusterMetrics incompatibility issues with MR1
 -

 Key: MAPREDUCE-5280
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5280
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Fix For: 2.1.0-beta

 Attachments: MAPREDUCE-5280-trunk-1.patch


 1. Constructor has one fewer parameters: numGraylistedTrackers
 2. getGrayListedTaskTrackerCount() is removed

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-5280) Mapreduce API: ClusterMetrics incompatibility issues with MR1

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13671973#comment-13671973
 ] 

Hudson commented on MAPREDUCE-5280:
---

Integrated in Hadoop-trunk-Commit #3826 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3826/])
MAPREDUCE-5280. Bring back removed constructor and a method in mapreduce 
ClusterMetrics for binary compatibility with 1.x APIs. Contributed by Mayank 
Bansal. (Revision 1488458)

 Result = SUCCESS
vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1488458
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ClusterStatus.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/ClusterMetrics.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestNetworkedJob.java


 Mapreduce API: ClusterMetrics incompatibility issues with MR1
 -

 Key: MAPREDUCE-5280
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5280
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
 Fix For: 2.1.0-beta

 Attachments: MAPREDUCE-5280-trunk-1.patch


 1. Constructor has one fewer parameters: numGraylistedTrackers
 2. getGrayListedTaskTrackerCount() is removed

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira