[jira] [Commented] (MAPREDUCE-3342) JobHistoryServer doesn't show job queue

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146130#comment-13146130
 ] 

Hudson commented on MAPREDUCE-3342:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #1278 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1278/])
MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue name. 
Contributed by Jonathan Eagles.

vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199133
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java


> JobHistoryServer doesn't show job queue
> ---
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3345) Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes

[Vinod Kumar Vavilapalli (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-3345:
---

  Resolution: Fixed
Release Note: Fixed a race condition in ResourceManager that was causing 
TestContainerManagerSecurity to fail sometimes.
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

I just committed this to trunk and branch-0.23. Thanks Hitesh!

> Race condition in ResourceManager causing TestContainerManagerSecurity to 
> fail sometimes
> 
>
> Key: MAPREDUCE-3345
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3345
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3345.1.patch
>
>
> See 
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1247//testReport/org.apache.hadoop.yarn.server/TestContainerManagerSecurity/testUnauthorizedUser/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3345) Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes

[Vinod Kumar Vavilapalli (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146128#comment-13146128
 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-3345:


I was wondering how to fix this till I saw your patch.

Patch is inline with my understanding of the race condition.

I'm fine with not having a test for this race. +1 for the patch.

> Race condition in ResourceManager causing TestContainerManagerSecurity to 
> fail sometimes
> 
>
> Key: MAPREDUCE-3345
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3345
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3345.1.patch
>
>
> See 
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1247//testReport/org.apache.hadoop.yarn.server/TestContainerManagerSecurity/testUnauthorizedUser/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) JobHistoryServer doesn't show job queue

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146129#comment-13146129
 ] 

Hudson commented on MAPREDUCE-3342:
---

Integrated in Hadoop-Mapreduce-0.23-Commit #166 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/166/])
MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue name. 
Contributed by Jonathan Eagles.
svn merge -c r1199133 --ignore-ancestry ../../trunk/

vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199135
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java


> JobHistoryServer doesn't show job queue
> ---
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3370) MiniMRYarnCluster uses a hard coded path location for the MapReduce application jar

[Vinod Kumar Vavilapalli (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-3370:
---

Fix Version/s: 0.23.1
   Status: Open  (was: Patch Available)

Can you also take care of MAPREDUCE-3338 as part of this? We need to make sure 
no test depends on MiniMRYarnCluster.HADOOP_MAPREDUCE_CLIENT_APP_JAR_NAME 
because of the hard-coded version. If you agree, please also fix 
TestDistributedShell to not use hard-coded version.

Also, how do you plan to set the sytem property? Need some changes to pom.xml 
files too?

> MiniMRYarnCluster uses a hard coded path location for the MapReduce 
> application jar
> ---
>
> Key: MAPREDUCE-3370
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3370
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, test
>Affects Versions: 0.23.0
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3370.patch
>
>
> MiniMRYarnCluster uses a hard coded relative path location for the MapReduce 
> application jar. It is better to have this location as a system property so 
> tests can pick the application jar regardless of their working directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) JobHistoryServer doesn't show job queue

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146122#comment-13146122
 ] 

Hudson commented on MAPREDUCE-3342:
---

Integrated in Hadoop-Common-trunk-Commit #1256 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1256/])
MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue name. 
Contributed by Jonathan Eagles.

vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199133
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java


> JobHistoryServer doesn't show job queue
> ---
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) JobHistoryServer doesn't show job queue

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146123#comment-13146123
 ] 

Hudson commented on MAPREDUCE-3342:
---

Integrated in Hadoop-Common-0.23-Commit #155 (See 
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/155/])
MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue name. 
Contributed by Jonathan Eagles.
svn merge -c r1199133 --ignore-ancestry ../../trunk/

vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199135
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java


> JobHistoryServer doesn't show job queue
> ---
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) JobHistoryServer doesn't show job queue

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146120#comment-13146120
 ] 

Hudson commented on MAPREDUCE-3342:
---

Integrated in Hadoop-Hdfs-0.23-Commit #154 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/154/])
MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue name. 
Contributed by Jonathan Eagles.
svn merge -c r1199133 --ignore-ancestry ../../trunk/

vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199135
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java


> JobHistoryServer doesn't show job queue
> ---
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) JobHistoryServer doesn't show job queue

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146121#comment-13146121
 ] 

Hudson commented on MAPREDUCE-3342:
---

Integrated in Hadoop-Hdfs-trunk-Commit #1330 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1330/])
MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue name. 
Contributed by Jonathan Eagles.

vinodkv : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199133
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java


> JobHistoryServer doesn't show job queue
> ---
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3342) JobHistoryServer doesn't show job queue

[Vinod Kumar Vavilapalli (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-3342:
---

Component/s: jobhistoryserver

> JobHistoryServer doesn't show job queue
> ---
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3342) JobHistoryServer doesn't show job queue

[Vinod Kumar Vavilapalli (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-3342:
---

  Resolution: Fixed
Release Note: Fixed JobHistoryServer to also show the job's queue name.
  Status: Resolved  (was: Patch Available)

Just committed this to trunk and branch-0.23. Thanks Jonathan!

> JobHistoryServer doesn't show job queue
> ---
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobhistoryserver, mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3342) JobHistoryServer doesn't show job queue

[Vinod Kumar Vavilapalli (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-3342:
---

Fix Version/s: 0.23.1
  Summary: JobHistoryServer doesn't show job queue  (was: job history 
doesn't show job queue)
 Hadoop Flags: Reviewed

Patch looks good to me too. +1.

> JobHistoryServer doesn't show job queue
> ---
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3121) NodeManager should handle disk-failures

[Vinod Kumar Vavilapalli (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated MAPREDUCE-3121:
---

Status: Open  (was: Patch Available)

My comments on the patch.

*Major:*
DiskHealthCheckerService
 - Let us not use the configuration property to read the dir-list anywhere. I 
already see a couple of places where this can be trouble-some. Make sure that 
YarnConfiguration.NM_LOCAL_DIRS and YarnConfiguration.NM_LOG_DIRS aren't read 
anywhere after the initial NM-start.
   -- After this, you should definitely drop 
DiskHealthCheckerService.updateDirsInConfiguration().
 - ResourceLocalizationService.localDirsSelector doesn't look at healthy disks, 
same for ShuffleHandler.
 - Let's have a single API to return local-dirs either Array or a list. Let the 
caller take care of any conversions they need. Ditto about 
getLocalDirsString()/getLogDirsString().
 - Same as above for the LocalStorage class and getNumDirs() methods in there.
 - Instead of setting the disksInfo in NodePage, we can set it directly in the 
health-report via NodeStatusUpdater. That way, RM will also get the report.

bq. IMO, one approach we could take is to do a check at the top level. For 
example, ContainersLaunch as per the patch does a sanity check on available 
disks and bails if there is an issue. It could create a snapshot of available 
disks at this point.
Agreed, we should check at ContainersLauncherImpl level before launching every 
container and send out a container-failed event in case all disks have gone bad.

*Minor:*
DiskHealthCheckerService
 - DisksHealthMonitorExecutor can be static and final.
 - Drop the constructor DiskHealthCheckerService(Configuration conf) and use 
init() life-cycle directly.
 - init(Configuration conf): call super.init() at the end so that it can set 
the service-state correctly. Same for start(), stop()
 - Declare diskHealthChecker as a DisksHealthMonitorExecutor instead of 
TimerTask so as to avoid unnecessary casting in get{Log|Local}Dirs()
 - LocalStorage is way too general. Let's rename LocalStorage to 
LocalDirectories?

Let's have multiple local and log-dirs by default in MiniYarnCluster.

TestDiskFailures:
 - No need for hard-coded 2 secs sleep, MiniYarnCluster already has code to 
wait for starting.
 - waitForDiskHealthCheck() is error-prone. You can set an inline 
DiskHealthChecker and invoke explicit checking of disks.

*Miscellaneous:*
Like I summarized in my first comment, there are more pending items. Let me 
know if you want to tackle them separately:
 - We need a configuration for the max-percentage of disks that can fail 
without shutting down NM.
 - If all or a major percentage of disks turn bad at startup, NodeManager 
should either fail to start or get started, and report the RM that it is bad 
and needs operator intervention.
 - If all or a major percentage of disks turn bad during runtime, NodeManager 
can update the RM and ask for a halt in the scheduling.


> NodeManager should handle disk-failures
> ---
>
> Key: MAPREDUCE-3121
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3121
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, nodemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Ravi Gummadi
> Fix For: 0.23.1
>
> Attachments: 3121.patch
>
>
> This is akin to MAPREDUCE-2413 but for YARN's NodeManager. We want to 
> minimize the impact of transient/permanent disk failures on containers. With 
> larger number of disks per node, the ability to continue to run containers on 
> other disks is crucial.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3343) TaskTracker Out of Memory because of distributed cache

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146095#comment-13146095
 ] 

Hadoop QA commented on MAPREDUCE-3343:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12502884/mapreduce-3343-release-0.20.205.0.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1269//console

This message is automatically generated.

> TaskTracker Out of Memory because of distributed cache
> --
>
> Key: MAPREDUCE-3343
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3343
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv1
>Affects Versions: 0.20.205.0
>Reporter: Ahmed Radwan
>  Labels: mapreduce, patch
> Attachments: mapreduce-3343-release-0.20.205.0.patch
>
>
> This Out of Memory happens when you run large number of jobs (using the 
> distributed cache) on a TaskTracker. 
> Seems the basic issue is with the distributedCacheManager (instance of 
> TrackerDistributedCacheManager in TaskTracker.java), this gets created during 
> TaskTracker.initialize(), and it keeps references to 
> TaskDistributedCacheManager for every submitted job via the jobArchives Map, 
> also references to CacheStatus via cachedArchives map. I am not seeing these 
> cleaned up between jobs, so this can out of memory problems after really 
> large number of jobs are submitted. We have seen this issue in a number of 
> cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-1976) Inconsistency in logging hostnames to JobHistory

[Amar Kamat (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-1976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amar Kamat resolved MAPREDUCE-1976.
---

Resolution: Duplicate

Duplicate of MAPREDUCE-3349.

> Inconsistency in logging hostnames to JobHistory
> 
>
> Key: MAPREDUCE-1976
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1976
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: jobtracker
>Reporter: Amar Kamat
>
> JobHistory line for successful task-attempts store tracker-name (i.e 
> /rack/xyz.com:1234) for the _hostname_ field while unsuccessful task-attempts 
> store only the hostname (i.e xyz.com).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3343) TaskTracker Out of Memory because of distributed cache

[zhaoyunjiong (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhaoyunjiong updated MAPREDUCE-3343:


Attachment: mapreduce-3343-release-0.20.205.0.patch

> TaskTracker Out of Memory because of distributed cache
> --
>
> Key: MAPREDUCE-3343
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3343
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv1
>Affects Versions: 0.20.205.0
>Reporter: Ahmed Radwan
>  Labels: mapreduce, patch
> Attachments: mapreduce-3343-release-0.20.205.0.patch
>
>
> This Out of Memory happens when you run large number of jobs (using the 
> distributed cache) on a TaskTracker. 
> Seems the basic issue is with the distributedCacheManager (instance of 
> TrackerDistributedCacheManager in TaskTracker.java), this gets created during 
> TaskTracker.initialize(), and it keeps references to 
> TaskDistributedCacheManager for every submitted job via the jobArchives Map, 
> also references to CacheStatus via cachedArchives map. I am not seeing these 
> cleaned up between jobs, so this can out of memory problems after really 
> large number of jobs are submitted. We have seen this issue in a number of 
> cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3343) TaskTracker Out of Memory because of distributed cache

[zhaoyunjiong (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhaoyunjiong updated MAPREDUCE-3343:


Attachment: (was: mapreduce-3343-release-0.20.205.0.patch)

> TaskTracker Out of Memory because of distributed cache
> --
>
> Key: MAPREDUCE-3343
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3343
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv1
>Affects Versions: 0.20.205.0
>Reporter: Ahmed Radwan
>  Labels: mapreduce, patch
> Attachments: mapreduce-3343-release-0.20.205.0.patch
>
>
> This Out of Memory happens when you run large number of jobs (using the 
> distributed cache) on a TaskTracker. 
> Seems the basic issue is with the distributedCacheManager (instance of 
> TrackerDistributedCacheManager in TaskTracker.java), this gets created during 
> TaskTracker.initialize(), and it keeps references to 
> TaskDistributedCacheManager for every submitted job via the jobArchives Map, 
> also references to CacheStatus via cachedArchives map. I am not seeing these 
> cleaned up between jobs, so this can out of memory problems after really 
> large number of jobs are submitted. We have seen this issue in a number of 
> cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3343) TaskTracker Out of Memory because of distributed cache

[zhaoyunjiong (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhaoyunjiong updated MAPREDUCE-3343:


Attachment: (was: 
bug-fix-avoid-memory-leak-in-TrackerDistributedCacheManager.patch)

> TaskTracker Out of Memory because of distributed cache
> --
>
> Key: MAPREDUCE-3343
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3343
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv1
>Affects Versions: 0.20.205.0
>Reporter: Ahmed Radwan
>  Labels: mapreduce, patch
> Attachments: mapreduce-3343-release-0.20.205.0.patch
>
>
> This Out of Memory happens when you run large number of jobs (using the 
> distributed cache) on a TaskTracker. 
> Seems the basic issue is with the distributedCacheManager (instance of 
> TrackerDistributedCacheManager in TaskTracker.java), this gets created during 
> TaskTracker.initialize(), and it keeps references to 
> TaskDistributedCacheManager for every submitted job via the jobArchives Map, 
> also references to CacheStatus via cachedArchives map. I am not seeing these 
> cleaned up between jobs, so this can out of memory problems after really 
> large number of jobs are submitted. We have seen this issue in a number of 
> cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3343) TaskTracker Out of Memory because of distributed cache

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146087#comment-13146087
 ] 

Hadoop QA commented on MAPREDUCE-3343:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12502883/mapreduce-3343-release-0.20.205.0.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1268//console

This message is automatically generated.

> TaskTracker Out of Memory because of distributed cache
> --
>
> Key: MAPREDUCE-3343
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3343
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv1
>Affects Versions: 0.20.205.0
>Reporter: Ahmed Radwan
>  Labels: mapreduce, patch
> Attachments: 
> bug-fix-avoid-memory-leak-in-TrackerDistributedCacheManager.patch, 
> mapreduce-3343-release-0.20.205.0.patch
>
>
> This Out of Memory happens when you run large number of jobs (using the 
> distributed cache) on a TaskTracker. 
> Seems the basic issue is with the distributedCacheManager (instance of 
> TrackerDistributedCacheManager in TaskTracker.java), this gets created during 
> TaskTracker.initialize(), and it keeps references to 
> TaskDistributedCacheManager for every submitted job via the jobArchives Map, 
> also references to CacheStatus via cachedArchives map. I am not seeing these 
> cleaned up between jobs, so this can out of memory problems after really 
> large number of jobs are submitted. We have seen this issue in a number of 
> cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3346) Rumen LoggedTaskAttempt getHostName call returns hostname as null

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146086#comment-13146086
 ] 

Hudson commented on MAPREDUCE-3346:
---

Integrated in Hadoop-Mapreduce-0.23-Commit #165 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/165/])
MAPREDUCE-3346. [Rumen] LoggedTaskAttempt#getHostName() returns null. 
(amarrk)

amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199106
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java


> Rumen LoggedTaskAttempt  getHostName call returns hostname as null
> --
>
> Key: MAPREDUCE-3346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3346
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: tools/rumen
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Amar Kamat
>Priority: Blocker
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3346-v1.1.patch
>
>
> After MAPREDUCE-3035 and MAPREDUCE-3317
> Now MRV2 job history contains hostName and rackName.
> when rumen trace builder is ran on jobhistory, its generated trace contains 
> hostname in form of 
> hostName : /raclname/hostname
> But getHostName for LoggedTaskAttempt returns hostname as null
> Seems that TraceBuilder is setting hostName properly but JobTraceReader is 
> not able read it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3343) TaskTracker Out of Memory because of distributed cache

[zhaoyunjiong (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhaoyunjiong updated MAPREDUCE-3343:


Attachment: mapreduce-3343-release-0.20.205.0.patch

Add a unit test. 
This patch works fine in our production cluster.

> TaskTracker Out of Memory because of distributed cache
> --
>
> Key: MAPREDUCE-3343
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3343
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv1
>Affects Versions: 0.20.205.0
>Reporter: Ahmed Radwan
>  Labels: mapreduce, patch
> Attachments: 
> bug-fix-avoid-memory-leak-in-TrackerDistributedCacheManager.patch, 
> mapreduce-3343-release-0.20.205.0.patch
>
>
> This Out of Memory happens when you run large number of jobs (using the 
> distributed cache) on a TaskTracker. 
> Seems the basic issue is with the distributedCacheManager (instance of 
> TrackerDistributedCacheManager in TaskTracker.java), this gets created during 
> TaskTracker.initialize(), and it keeps references to 
> TaskDistributedCacheManager for every submitted job via the jobArchives Map, 
> also references to CacheStatus via cachedArchives map. I am not seeing these 
> cleaned up between jobs, so this can out of memory problems after really 
> large number of jobs are submitted. We have seen this issue in a number of 
> cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3346) Rumen LoggedTaskAttempt getHostName call returns hostname as null

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146084#comment-13146084
 ] 

Hudson commented on MAPREDUCE-3346:
---

Integrated in Hadoop-Common-0.23-Commit #154 (See 
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/154/])
MAPREDUCE-3346. [Rumen] LoggedTaskAttempt#getHostName() returns null. 
(amarrk)

amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199106
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java


> Rumen LoggedTaskAttempt  getHostName call returns hostname as null
> --
>
> Key: MAPREDUCE-3346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3346
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: tools/rumen
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Amar Kamat
>Priority: Blocker
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3346-v1.1.patch
>
>
> After MAPREDUCE-3035 and MAPREDUCE-3317
> Now MRV2 job history contains hostName and rackName.
> when rumen trace builder is ran on jobhistory, its generated trace contains 
> hostname in form of 
> hostName : /raclname/hostname
> But getHostName for LoggedTaskAttempt returns hostname as null
> Seems that TraceBuilder is setting hostName properly but JobTraceReader is 
> not able read it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3346) Rumen LoggedTaskAttempt getHostName call returns hostname as null

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146081#comment-13146081
 ] 

Hudson commented on MAPREDUCE-3346:
---

Integrated in Hadoop-Hdfs-0.23-Commit #153 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/153/])
MAPREDUCE-3346. [Rumen] LoggedTaskAttempt#getHostName() returns null. 
(amarrk)

amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1199106
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/tools/org/apache/hadoop/tools/rumen/LoggedTaskAttempt.java


> Rumen LoggedTaskAttempt  getHostName call returns hostname as null
> --
>
> Key: MAPREDUCE-3346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3346
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: tools/rumen
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Amar Kamat
>Priority: Blocker
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3346-v1.1.patch
>
>
> After MAPREDUCE-3035 and MAPREDUCE-3317
> Now MRV2 job history contains hostName and rackName.
> when rumen trace builder is ran on jobhistory, its generated trace contains 
> hostname in form of 
> hostName : /raclname/hostname
> But getHostName for LoggedTaskAttempt returns hostname as null
> Seems that TraceBuilder is setting hostName properly but JobTraceReader is 
> not able read it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-3346) Rumen LoggedTaskAttempt getHostName call returns hostname as null

[Amar Kamat (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amar Kamat resolved MAPREDUCE-3346.
---

Tags: rumen hostname 
  Resolution: Fixed
   Fix Version/s: 0.23.1
  0.24.0
Target Version/s: 0.24.0, 0.23.1

I just committed this to trunk and branch-0.23. Thanks Karam!

> Rumen LoggedTaskAttempt  getHostName call returns hostname as null
> --
>
> Key: MAPREDUCE-3346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3346
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: tools/rumen
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Amar Kamat
>Priority: Blocker
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3346-v1.1.patch
>
>
> After MAPREDUCE-3035 and MAPREDUCE-3317
> Now MRV2 job history contains hostName and rackName.
> when rumen trace builder is ran on jobhistory, its generated trace contains 
> hostname in form of 
> hostName : /raclname/hostname
> But getHostName for LoggedTaskAttempt returns hostname as null
> Seems that TraceBuilder is setting hostName properly but JobTraceReader is 
> not able read it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3346) Rumen LoggedTaskAttempt getHostName call returns hostname as null

[Amar Kamat (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amar Kamat updated MAPREDUCE-3346:
--

Attachment: MAPREDUCE-3346-v1.1.patch

> Rumen LoggedTaskAttempt  getHostName call returns hostname as null
> --
>
> Key: MAPREDUCE-3346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3346
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: tools/rumen
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Amar Kamat
>Priority: Blocker
> Attachments: MAPREDUCE-3346-v1.1.patch
>
>
> After MAPREDUCE-3035 and MAPREDUCE-3317
> Now MRV2 job history contains hostName and rackName.
> when rumen trace builder is ran on jobhistory, its generated trace contains 
> hostname in form of 
> hostName : /raclname/hostname
> But getHostName for LoggedTaskAttempt returns hostname as null
> Seems that TraceBuilder is setting hostName properly but JobTraceReader is 
> not able read it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3346) Rumen LoggedTaskAttempt getHostName call returns hostname as null

[Amar Kamat (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amar Kamat updated MAPREDUCE-3346:
--

Attachment: (was: BZ-4980655-v1.1.patch)

> Rumen LoggedTaskAttempt  getHostName call returns hostname as null
> --
>
> Key: MAPREDUCE-3346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3346
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: tools/rumen
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Amar Kamat
>Priority: Blocker
>
> After MAPREDUCE-3035 and MAPREDUCE-3317
> Now MRV2 job history contains hostName and rackName.
> when rumen trace builder is ran on jobhistory, its generated trace contains 
> hostname in form of 
> hostName : /raclname/hostname
> But getHostName for LoggedTaskAttempt returns hostname as null
> Seems that TraceBuilder is setting hostName properly but JobTraceReader is 
> not able read it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3346) Rumen LoggedTaskAttempt getHostName call returns hostname as null

[Amar Kamat (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146074#comment-13146074
 ] 

Amar Kamat commented on MAPREDUCE-3346:
---

The corner case should be fixed in MAPREDUCE-1976. I will commit this patch.

> Rumen LoggedTaskAttempt  getHostName call returns hostname as null
> --
>
> Key: MAPREDUCE-3346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3346
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: tools/rumen
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Amar Kamat
>Priority: Blocker
>
> After MAPREDUCE-3035 and MAPREDUCE-3317
> Now MRV2 job history contains hostName and rackName.
> when rumen trace builder is ran on jobhistory, its generated trace contains 
> hostname in form of 
> hostName : /raclname/hostname
> But getHostName for LoggedTaskAttempt returns hostname as null
> Seems that TraceBuilder is setting hostName properly but JobTraceReader is 
> not able read it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3365) Uncomment eventlog settings from the documentation

[Harsh J (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-3365:
---

 Tags: docs
   Resolution: Fixed
Fix Version/s: (was: 0.20.205.0)
   0.20.206.0
   Status: Resolved  (was: Patch Available)

Committed as revision 1199059.

Thanks you for your contribution Sho! :)

> Uncomment eventlog settings from the documentation
> --
>
> Key: MAPREDUCE-3365
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3365
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: contrib/fair-share
>Affects Versions: 0.20.205.0
>Reporter: Sho Shimauchi
>Assignee: Sho Shimauchi
>Priority: Trivial
> Fix For: 0.20.206.0
>
> Attachments: MAPREDUCE-3365.patch
>
>
> Two fair scheduler debug options "mapred.fairscheduler.eventlog.enabled" and 
> "mapred.fairscheduler.dump.interval" are commented out in fair scheduler doc 
> file.
> It's useful for debugging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3374) src/c++/task-controller/configure is not set executable in the tarball and that prevents task-controller from rebuilding

[Matt Foley (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146037#comment-13146037
 ] 

Matt Foley commented on MAPREDUCE-3374:
---

Hi Roman, thanks for contributing this patch.  Please run test-patch manually 
against branch-0.20-security-205, and paste in the result evaluation as a 
comment in this Jira. Thanks.

> src/c++/task-controller/configure is not set executable in the tarball and 
> that prevents task-controller from rebuilding
> 
>
> Key: MAPREDUCE-3374
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3374
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: task-controller
>Affects Versions: 0.20.205.0
>Reporter: Roman Shaposhnik
> Attachments: MAPREDUCE-3374.patch.txt
>
>
> ant task-controller fails because src/c++/task-controller/configure is not 
> set executable

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3365) Uncomment eventlog settings from the documentation

[Harsh J (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146036#comment-13146036
 ] 

Harsh J commented on MAPREDUCE-3365:


Ack, I keep forgetting that we do not have target version builds. Patch 
obviously doesn't apply to trunk.

Tested patch locally. Documentation builds fine and the result HTML looks good. 
Committing.

> Uncomment eventlog settings from the documentation
> --
>
> Key: MAPREDUCE-3365
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3365
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: contrib/fair-share
>Affects Versions: 0.20.205.0
>Reporter: Sho Shimauchi
>Assignee: Sho Shimauchi
>Priority: Trivial
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-3365.patch
>
>
> Two fair scheduler debug options "mapred.fairscheduler.eventlog.enabled" and 
> "mapred.fairscheduler.dump.interval" are commented out in fair scheduler doc 
> file.
> It's useful for debugging.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3374) src/c++/task-controller/configure is not set executable in the tarball and that prevents task-controller from rebuilding

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146025#comment-13146025
 ] 

Hadoop QA commented on MAPREDUCE-3374:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12502860/MAPREDUCE-3374.patch.txt
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1267//console

This message is automatically generated.

> src/c++/task-controller/configure is not set executable in the tarball and 
> that prevents task-controller from rebuilding
> 
>
> Key: MAPREDUCE-3374
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3374
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: task-controller
>Affects Versions: 0.20.205.0
>Reporter: Roman Shaposhnik
> Attachments: MAPREDUCE-3374.patch.txt
>
>
> ant task-controller fails because src/c++/task-controller/configure is not 
> set executable

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3323) Add new interface for Distributed Cache, which special for Map or Reduce,but not Both.

[Azuryy(Chijiong) (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Azuryy(Chijiong) updated MAPREDUCE-3323:


Attachment: (was: DistributedCache.patch)

> Add new interface for Distributed Cache, which special  for Map or Reduce,but 
> not Both.
> ---
>
> Key: MAPREDUCE-3323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache, tasktracker
>Affects Versions: 0.20.203.0
>Reporter: Azuryy(Chijiong)
> Fix For: 0.20.203.0
>
>
> We put some file into Distributed Cache, but sometimes, only Map or Reduce 
> use thses cached files, not useful for both. but TaskTracker always download 
> cached files from HDFS, if there are some little bit big files in cache, it's 
> time expensive.
> so, this patch add some new API in the DistributedCache.java as follow:
> addArchiveToClassPathForMap
> addArchiveToClassPathForReduce
> addFileToClassPathForMap
> addFileToClassPathForReduce
> addCacheFileForMap
> addCacheFileForReduce
> addCacheArchiveForMap
> addCacheArchiveForReduce
> New API doesn't affect original interface. User can use these features like 
> the following two methods:
> 1) 
> hadoop job  -files file1 -mapfiles file2 -reducefiles file3 -archives 
> arc1 -maparchives arc2 -reduce archives arc3
> 2)
> DistributedCache.addCacheFile(conf, file1);
> DistributedCache.addCacheFileForMap(conf, file2);
> DistributedCache.addCacheFileForReduce(conf, file3);
> DistributedCache.addCacheArchives(conf, arc1);
> DistributedCache.addCacheArchivesForMap(conf, arc2);
> DistributedCache.addCacheFArchivesForReduce(conf, arc3);
> These two methods have the same result, That's mean: 
> You put six files to the distributed cache: file1 ~ file3, arc1 ~ arc3, 
> but file1 and arc1 are cached for both map and reduce;
> file2 and arc2 are only cached for map;
> file3 and arc3 are only cached for reduce;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3323) Add new interface for Distributed Cache, which special for Map or Reduce,but not Both.

[Azuryy(Chijiong) (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Azuryy(Chijiong) updated MAPREDUCE-3323:


Attachment: (was: GenericOptionsParser.patch)

> Add new interface for Distributed Cache, which special  for Map or Reduce,but 
> not Both.
> ---
>
> Key: MAPREDUCE-3323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache, tasktracker
>Affects Versions: 0.20.203.0
>Reporter: Azuryy(Chijiong)
> Fix For: 0.20.203.0
>
>
> We put some file into Distributed Cache, but sometimes, only Map or Reduce 
> use thses cached files, not useful for both. but TaskTracker always download 
> cached files from HDFS, if there are some little bit big files in cache, it's 
> time expensive.
> so, this patch add some new API in the DistributedCache.java as follow:
> addArchiveToClassPathForMap
> addArchiveToClassPathForReduce
> addFileToClassPathForMap
> addFileToClassPathForReduce
> addCacheFileForMap
> addCacheFileForReduce
> addCacheArchiveForMap
> addCacheArchiveForReduce
> New API doesn't affect original interface. User can use these features like 
> the following two methods:
> 1) 
> hadoop job  -files file1 -mapfiles file2 -reducefiles file3 -archives 
> arc1 -maparchives arc2 -reduce archives arc3
> 2)
> DistributedCache.addCacheFile(conf, file1);
> DistributedCache.addCacheFileForMap(conf, file2);
> DistributedCache.addCacheFileForReduce(conf, file3);
> DistributedCache.addCacheArchives(conf, arc1);
> DistributedCache.addCacheArchivesForMap(conf, arc2);
> DistributedCache.addCacheFArchivesForReduce(conf, arc3);
> These two methods have the same result, That's mean: 
> You put six files to the distributed cache: file1 ~ file3, arc1 ~ arc3, 
> but file1 and arc1 are cached for both map and reduce;
> file2 and arc2 are only cached for map;
> file3 and arc3 are only cached for reduce;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3323) Add new interface for Distributed Cache, which special for Map or Reduce,but not Both.

[Azuryy(Chijiong) (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Azuryy(Chijiong) updated MAPREDUCE-3323:


Attachment: (was: JobClient.patch)

> Add new interface for Distributed Cache, which special  for Map or Reduce,but 
> not Both.
> ---
>
> Key: MAPREDUCE-3323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache, tasktracker
>Affects Versions: 0.20.203.0
>Reporter: Azuryy(Chijiong)
> Fix For: 0.20.203.0
>
>
> We put some file into Distributed Cache, but sometimes, only Map or Reduce 
> use thses cached files, not useful for both. but TaskTracker always download 
> cached files from HDFS, if there are some little bit big files in cache, it's 
> time expensive.
> so, this patch add some new API in the DistributedCache.java as follow:
> addArchiveToClassPathForMap
> addArchiveToClassPathForReduce
> addFileToClassPathForMap
> addFileToClassPathForReduce
> addCacheFileForMap
> addCacheFileForReduce
> addCacheArchiveForMap
> addCacheArchiveForReduce
> New API doesn't affect original interface. User can use these features like 
> the following two methods:
> 1) 
> hadoop job  -files file1 -mapfiles file2 -reducefiles file3 -archives 
> arc1 -maparchives arc2 -reduce archives arc3
> 2)
> DistributedCache.addCacheFile(conf, file1);
> DistributedCache.addCacheFileForMap(conf, file2);
> DistributedCache.addCacheFileForReduce(conf, file3);
> DistributedCache.addCacheArchives(conf, arc1);
> DistributedCache.addCacheArchivesForMap(conf, arc2);
> DistributedCache.addCacheFArchivesForReduce(conf, arc3);
> These two methods have the same result, That's mean: 
> You put six files to the distributed cache: file1 ~ file3, arc1 ~ arc3, 
> but file1 and arc1 are cached for both map and reduce;
> file2 and arc2 are only cached for map;
> file3 and arc3 are only cached for reduce;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3323) Add new interface for Distributed Cache, which special for Map or Reduce,but not Both.

[Azuryy(Chijiong) (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Azuryy(Chijiong) updated MAPREDUCE-3323:


Attachment: (was: TaskTracker.patch)

> Add new interface for Distributed Cache, which special  for Map or Reduce,but 
> not Both.
> ---
>
> Key: MAPREDUCE-3323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache, tasktracker
>Affects Versions: 0.20.203.0
>Reporter: Azuryy(Chijiong)
> Fix For: 0.20.203.0
>
>
> We put some file into Distributed Cache, but sometimes, only Map or Reduce 
> use thses cached files, not useful for both. but TaskTracker always download 
> cached files from HDFS, if there are some little bit big files in cache, it's 
> time expensive.
> so, this patch add some new API in the DistributedCache.java as follow:
> addArchiveToClassPathForMap
> addArchiveToClassPathForReduce
> addFileToClassPathForMap
> addFileToClassPathForReduce
> addCacheFileForMap
> addCacheFileForReduce
> addCacheArchiveForMap
> addCacheArchiveForReduce
> New API doesn't affect original interface. User can use these features like 
> the following two methods:
> 1) 
> hadoop job  -files file1 -mapfiles file2 -reducefiles file3 -archives 
> arc1 -maparchives arc2 -reduce archives arc3
> 2)
> DistributedCache.addCacheFile(conf, file1);
> DistributedCache.addCacheFileForMap(conf, file2);
> DistributedCache.addCacheFileForReduce(conf, file3);
> DistributedCache.addCacheArchives(conf, arc1);
> DistributedCache.addCacheArchivesForMap(conf, arc2);
> DistributedCache.addCacheFArchivesForReduce(conf, arc3);
> These two methods have the same result, That's mean: 
> You put six files to the distributed cache: file1 ~ file3, arc1 ~ arc3, 
> but file1 and arc1 are cached for both map and reduce;
> file2 and arc2 are only cached for map;
> file3 and arc3 are only cached for reduce;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3323) Add new interface for Distributed Cache, which special for Map or Reduce,but not Both.

[Azuryy(Chijiong) (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Azuryy(Chijiong) updated MAPREDUCE-3323:


Attachment: (was: TaskDistributedCacheManager.patch)

> Add new interface for Distributed Cache, which special  for Map or Reduce,but 
> not Both.
> ---
>
> Key: MAPREDUCE-3323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache, tasktracker
>Affects Versions: 0.20.203.0
>Reporter: Azuryy(Chijiong)
> Fix For: 0.20.203.0
>
>
> We put some file into Distributed Cache, but sometimes, only Map or Reduce 
> use thses cached files, not useful for both. but TaskTracker always download 
> cached files from HDFS, if there are some little bit big files in cache, it's 
> time expensive.
> so, this patch add some new API in the DistributedCache.java as follow:
> addArchiveToClassPathForMap
> addArchiveToClassPathForReduce
> addFileToClassPathForMap
> addFileToClassPathForReduce
> addCacheFileForMap
> addCacheFileForReduce
> addCacheArchiveForMap
> addCacheArchiveForReduce
> New API doesn't affect original interface. User can use these features like 
> the following two methods:
> 1) 
> hadoop job  -files file1 -mapfiles file2 -reducefiles file3 -archives 
> arc1 -maparchives arc2 -reduce archives arc3
> 2)
> DistributedCache.addCacheFile(conf, file1);
> DistributedCache.addCacheFileForMap(conf, file2);
> DistributedCache.addCacheFileForReduce(conf, file3);
> DistributedCache.addCacheArchives(conf, arc1);
> DistributedCache.addCacheArchivesForMap(conf, arc2);
> DistributedCache.addCacheFArchivesForReduce(conf, arc3);
> These two methods have the same result, That's mean: 
> You put six files to the distributed cache: file1 ~ file3, arc1 ~ arc3, 
> but file1 and arc1 are cached for both map and reduce;
> file2 and arc2 are only cached for map;
> file3 and arc3 are only cached for reduce;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3374) src/c++/task-controller/configure is not set executable in the tarball and that prevents task-controller from rebuilding

[Roman Shaposhnik (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Shaposhnik updated MAPREDUCE-3374:


Attachment: MAPREDUCE-3374.patch.txt

> src/c++/task-controller/configure is not set executable in the tarball and 
> that prevents task-controller from rebuilding
> 
>
> Key: MAPREDUCE-3374
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3374
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: task-controller
>Affects Versions: 0.20.205.0
>Reporter: Roman Shaposhnik
> Attachments: MAPREDUCE-3374.patch.txt
>
>
> ant task-controller fails because src/c++/task-controller/configure is not 
> set executable

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3374) src/c++/task-controller/configure is not set executable in the tarball and that prevents task-controller from rebuilding

[Roman Shaposhnik (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Shaposhnik updated MAPREDUCE-3374:


Status: Patch Available  (was: Open)

> src/c++/task-controller/configure is not set executable in the tarball and 
> that prevents task-controller from rebuilding
> 
>
> Key: MAPREDUCE-3374
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3374
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: task-controller
>Affects Versions: 0.20.205.0
>Reporter: Roman Shaposhnik
> Attachments: MAPREDUCE-3374.patch.txt
>
>
> ant task-controller fails because src/c++/task-controller/configure is not 
> set executable

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3369) Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169

[Tom White (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146014#comment-13146014
 ] 

Tom White commented on MAPREDUCE-3369:
--

This looks good. It would easier to review if you had a script to do a 'svn mv' 
followed by a patch generated with 'svn diff --no-diff-deleted 
--notice-ancestry'. We will need this anyway to preserve history when it gets 
committed. (Note that Jenkins can't apply move scripts, so you might want to 
post both types.)

Are you planning to port all the tests? I think it would be fine to port tests 
that need little or no modification to make them pass, and leave the ones that 
need more work to a follow-up JIRA.

> Migrate MR1 tests to run on MR2 using the new interfaces introduced in 
> MAPREDUCE-3169
> -
>
> Key: MAPREDUCE-3369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3369
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2, test
>Affects Versions: 0.23.0
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3369.patch
>
>
> This ticket tracks the migration of MR1 tests (currently residing in 
> "hadoop-mapreduce-project/src/test/") to run on MR2. The migration is using 
> the new interfaces introduced in MAPREDUCE-3169.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3323) Add new interface for Distributed Cache, which special for Map or Reduce,but not Both.

[Azuryy(Chijiong) (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146007#comment-13146007
 ] 

Azuryy(Chijiong) commented on MAPREDUCE-3323:
-

Thanks for all comments.

I've changed from patch available to unresolved. all patches will be optimized 
as your comments. 

> Add new interface for Distributed Cache, which special  for Map or Reduce,but 
> not Both.
> ---
>
> Key: MAPREDUCE-3323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache, tasktracker
>Affects Versions: 0.20.203.0
>Reporter: Azuryy(Chijiong)
> Fix For: 0.20.203.0
>
> Attachments: DistributedCache.patch, GenericOptionsParser.patch, 
> JobClient.patch, TaskDistributedCacheManager.patch, TaskTracker.patch
>
>
> We put some file into Distributed Cache, but sometimes, only Map or Reduce 
> use thses cached files, not useful for both. but TaskTracker always download 
> cached files from HDFS, if there are some little bit big files in cache, it's 
> time expensive.
> so, this patch add some new API in the DistributedCache.java as follow:
> addArchiveToClassPathForMap
> addArchiveToClassPathForReduce
> addFileToClassPathForMap
> addFileToClassPathForReduce
> addCacheFileForMap
> addCacheFileForReduce
> addCacheArchiveForMap
> addCacheArchiveForReduce
> New API doesn't affect original interface. User can use these features like 
> the following two methods:
> 1) 
> hadoop job  -files file1 -mapfiles file2 -reducefiles file3 -archives 
> arc1 -maparchives arc2 -reduce archives arc3
> 2)
> DistributedCache.addCacheFile(conf, file1);
> DistributedCache.addCacheFileForMap(conf, file2);
> DistributedCache.addCacheFileForReduce(conf, file3);
> DistributedCache.addCacheArchives(conf, arc1);
> DistributedCache.addCacheArchivesForMap(conf, arc2);
> DistributedCache.addCacheFArchivesForReduce(conf, arc3);
> These two methods have the same result, That's mean: 
> You put six files to the distributed cache: file1 ~ file3, arc1 ~ arc3, 
> but file1 and arc1 are cached for both map and reduce;
> file2 and arc2 are only cached for map;
> file3 and arc3 are only cached for reduce;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3374) src/c++/task-controller/configure is not set executable in the tarball and that prevents task-controller from rebuilding

[Roman Shaposhnik (Created) (JIRA)]

src/c++/task-controller/configure is not set executable in the tarball and that 
prevents task-controller from rebuilding


 Key: MAPREDUCE-3374
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3374
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: task-controller
Affects Versions: 0.20.205.0
Reporter: Roman Shaposhnik


ant task-controller fails because src/c++/task-controller/configure is not set 
executable

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146000#comment-13146000
 ] 

Hadoop QA commented on MAPREDUCE-3169:
--

+1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12502804/MAPREDUCE-3169-trunk_rev2.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 10 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1265//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1265//console

This message is automatically generated.

> Create a new MiniMRCluster equivalent which only provides client APIs cross 
> MR1 and MR2
> ---
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2, test
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, 
> MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster 
> for writing tests. Many users do as well. MiniMRCluster, however, exposes MR 
> implementation details like the existence of TaskTrackers, JobTrackers, etc, 
> since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by 
> MR1 or MR2 that exposes only the client-side portions of the MR framework. 
> Ideally it would be "recompile-compatible" with MiniMRCluster for most 
> applications, and the MR1 implementation could be backported to 20x branch. 
> Thus, dependent projects like HBase could migrate to this implementation and 
> test against both MR1 and MR2. We can also use this to port over the current 
> functional tests that use only the client-side features of MiniMRCluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3370) MiniMRYarnCluster uses a hard coded path location for the MapReduce application jar

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145999#comment-13145999
 ] 

Hadoop QA commented on MAPREDUCE-3370:
--

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12502800/MAPREDUCE-3370.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1264//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1264//console

This message is automatically generated.

> MiniMRYarnCluster uses a hard coded path location for the MapReduce 
> application jar
> ---
>
> Key: MAPREDUCE-3370
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3370
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, test
>Affects Versions: 0.23.0
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3370.patch
>
>
> MiniMRYarnCluster uses a hard coded relative path location for the MapReduce 
> application jar. It is better to have this location as a system property so 
> tests can pick the application jar regardless of their working directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3373) hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145998#comment-13145998
 ] 

Hadoop QA commented on MAPREDUCE-3373:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12502851/MAPREDUCE-3373.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1263//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1263//console

This message is automatically generated.

> hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh
> ---
>
> Key: MAPREDUCE-3373
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.23.0, 0.23.1
>Reporter: Bruno Mahé
>Assignee: Bruno Mahé
>  Labels: bigtop
> Attachments: MAPREDUCE-3373.patch
>
>
> It would be nice to be able to specify some other location for 
> hadoop-config.sh

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3369) Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145992#comment-13145992
 ] 

Hadoop QA commented on MAPREDUCE-3369:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12502803/MAPREDUCE-3369.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 100 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

-1 javac.  The patch appears to cause tar ant target to fail.

-1 findbugs.  The patch appears to cause Findbugs (version 1.3.9) to fail.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

-1 core tests.  The patch failed the unit tests build

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1266//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1266//console

This message is automatically generated.

> Migrate MR1 tests to run on MR2 using the new interfaces introduced in 
> MAPREDUCE-3169
> -
>
> Key: MAPREDUCE-3369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3369
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2, test
>Affects Versions: 0.23.0
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3369.patch
>
>
> This ticket tracks the migration of MR1 tests (currently residing in 
> "hadoop-mapreduce-project/src/test/") to run on MR2. The migration is using 
> the new interfaces introduced in MAPREDUCE-3169.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3373) hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh

[Updated]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Mahé updated MAPREDUCE-3373:
--

Attachment: MAPREDUCE-3373.patch

Here is a patch

> hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh
> ---
>
> Key: MAPREDUCE-3373
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.23.0, 0.23.1
>Reporter: Bruno Mahé
>Assignee: Bruno Mahé
>  Labels: bigtop
> Attachments: MAPREDUCE-3373.patch
>
>
> It would be nice to be able to specify some other location for 
> hadoop-config.sh

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3373) hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh

[Created]

hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh
---

 Key: MAPREDUCE-3373
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Bruno Mahé
 Attachments: MAPREDUCE-3373.patch

It would be nice to be able to specify some other location for hadoop-config.sh

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3373) hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh

[Updated]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Mahé updated MAPREDUCE-3373:
--

Status: Patch Available  (was: Open)

> hadoop script unconditionnaly source "$bin"/../libexec/hadoop-config.sh
> ---
>
> Key: MAPREDUCE-3373
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3373
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.23.0, 0.23.1
>Reporter: Bruno Mahé
>Assignee: Bruno Mahé
>  Labels: bigtop
> Attachments: MAPREDUCE-3373.patch
>
>
> It would be nice to be able to specify some other location for 
> hadoop-config.sh

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2

[Ahmed Radwan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-3169:


Target Version/s: 0.20.206.0, 0.23.0  (was: 0.23.0, 0.20.206.0)
  Status: Patch Available  (was: Open)

> Create a new MiniMRCluster equivalent which only provides client APIs cross 
> MR1 and MR2
> ---
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2, test
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, 
> MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster 
> for writing tests. Many users do as well. MiniMRCluster, however, exposes MR 
> implementation details like the existence of TaskTrackers, JobTrackers, etc, 
> since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by 
> MR1 or MR2 that exposes only the client-side portions of the MR framework. 
> Ideally it would be "recompile-compatible" with MiniMRCluster for most 
> applications, and the MR1 implementation could be backported to 20x branch. 
> Thus, dependent projects like HBase could migrate to this implementation and 
> test against both MR1 and MR2. We can also use this to port over the current 
> functional tests that use only the client-side features of MiniMRCluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3369) Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169

[Ahmed Radwan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-3369:


Status: Patch Available  (was: Open)

> Migrate MR1 tests to run on MR2 using the new interfaces introduced in 
> MAPREDUCE-3169
> -
>
> Key: MAPREDUCE-3369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3369
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2, test
>Affects Versions: 0.23.0
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3369.patch
>
>
> This ticket tracks the migration of MR1 tests (currently residing in 
> "hadoop-mapreduce-project/src/test/") to run on MR2. The migration is using 
> the new interfaces introduced in MAPREDUCE-3169.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3345) Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145971#comment-13145971
 ] 

Hadoop QA commented on MAPREDUCE-3345:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12502834/MR-3345.1.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1262//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1262//console

This message is automatically generated.

> Race condition in ResourceManager causing TestContainerManagerSecurity to 
> fail sometimes
> 
>
> Key: MAPREDUCE-3345
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3345
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3345.1.patch
>
>
> See 
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1247//testReport/org.apache.hadoop.yarn.server/TestContainerManagerSecurity/testUnauthorizedUser/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3370) MiniMRYarnCluster uses a hard coded path location for the MapReduce application jar

[Ahmed Radwan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-3370:


Status: Patch Available  (was: Open)

> MiniMRYarnCluster uses a hard coded path location for the MapReduce 
> application jar
> ---
>
> Key: MAPREDUCE-3370
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3370
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, test
>Affects Versions: 0.23.0
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3370.patch
>
>
> MiniMRYarnCluster uses a hard coded relative path location for the MapReduce 
> application jar. It is better to have this location as a system property so 
> tests can pick the application jar regardless of their working directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3371) Review and improve the yarn-api javadocs.

[Ravi Prakash (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated MAPREDUCE-3371:


Attachment: MAPREDUCE-3371.branch-0.23.patch

Please review these changes.

Also these points to note:
* Javadocs for  FinishApplicationMasterRequest and 
org.apache.hadoop.yarn.api.records.ContainerState aren’t being generated.
* AllocateRequest:getReleaseList() : What if the container is still being used? 
i.e. it is not unused as specified in the Javadoc?
* RegisterApplicationMasterResponse : Minimum and Maximum capabilities. As 
relevant to this user? Or cluster wide?
* StartContainerResponse.java: Please check my javadocs. Should I mention that 
shuffle is implemented as an auxiliary service? Also, how is the input 
ByteBuffer distinguished from the output ByteBuffer?
* QueueUserACLInfo : Who’s the given user? Do we mean current user?
* ContainerId: If the container is not assigned to an ApplicationAttemptId, 
what does getApplicationAttemptId() return?



> Review and improve the yarn-api javadocs.
> -
>
> Key: MAPREDUCE-3371
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3371
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: documentation, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
>Priority: Minor
>  Labels: documentation
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3371.branch-0.23.patch
>
>
> Review and improve the yarn-api javadocs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145916#comment-13145916
 ] 

Hadoop QA commented on MAPREDUCE-3342:
--

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12502827/MAPREDUCE-3342.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 9 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1261//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1261//console

This message is automatically generated.

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3345) Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes

[Hitesh Shah (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah updated MAPREDUCE-3345:
---

Status: Patch Available  (was: Open)

> Race condition in ResourceManager causing TestContainerManagerSecurity to 
> fail sometimes
> 
>
> Key: MAPREDUCE-3345
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3345
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3345.1.patch
>
>
> See 
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1247//testReport/org.apache.hadoop.yarn.server/TestContainerManagerSecurity/testUnauthorizedUser/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145915#comment-13145915
 ] 

Hadoop QA commented on MAPREDUCE-3372:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12502830/MAPREDUCE-3372.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1260//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1260//console

This message is automatically generated.

> HADOOP_PREFIX cannot be overriden
> -
>
> Key: MAPREDUCE-3372
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.23.0, 0.23.1
>Reporter: Bruno Mahé
>  Labels: bigtop
> Attachments: MAPREDUCE-3372.patch
>
>
> hadoop-config.sh forces HADOOP_prefix to a specific value:
> export HADOOP_PREFIX=`dirname "$this"`/..
> It would be nice to make this overridable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3345) Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes

[Hitesh Shah (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hitesh Shah updated MAPREDUCE-3345:
---

Attachment: MR-3345.1.patch

No tests - trivial fix. Running failing test in a loop for over an hour now 
with no failures. 

> Race condition in ResourceManager causing TestContainerManagerSecurity to 
> fail sometimes
> 
>
> Key: MAPREDUCE-3345
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3345
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Hitesh Shah
> Fix For: 0.23.1
>
> Attachments: MR-3345.1.patch
>
>
> See 
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1247//testReport/org.apache.hadoop.yarn.server/TestContainerManagerSecurity/testUnauthorizedUser/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3345) Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes

[Hitesh Shah (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145911#comment-13145911
 ] 

Hitesh Shah commented on MAPREDUCE-3345:


Debug logs seem to be point to a container being allocated for the AM in 
RMAppAttemptImpl#ScheduleTransition itself  ( when transitioning from SUBMITTED 
-> SCHEDULED ) which results in no allocated containers in the response when 
allocate is called subsequently in AMContainerAllocatedTransition. 

FifoScheduler ( unlike Capacity scheduler ) does not have a synchronized block 
in Scheduler#allocate per application which have otherwise prevented the above 
from happening.


> Race condition in ResourceManager causing TestContainerManagerSecurity to 
> fail sometimes
> 
>
> Key: MAPREDUCE-3345
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3345
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Hitesh Shah
> Fix For: 0.23.1
>
>
> See 
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1247//testReport/org.apache.hadoop.yarn.server/TestContainerManagerSecurity/testUnauthorizedUser/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden

[Updated]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Mahé updated MAPREDUCE-3372:
--

Attachment: MAPREDUCE-3372.patch

Here is a patch

> HADOOP_PREFIX cannot be overriden
> -
>
> Key: MAPREDUCE-3372
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.23.0, 0.23.1
>Reporter: Bruno Mahé
>  Labels: bigtop
> Attachments: MAPREDUCE-3372.patch
>
>
> hadoop-config.sh forces HADOOP_prefix to a specific value:
> export HADOOP_PREFIX=`dirname "$this"`/..
> It would be nice to make this overridable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden

[Updated]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Mahé updated MAPREDUCE-3372:
--

Status: Patch Available  (was: Open)

> HADOOP_PREFIX cannot be overriden
> -
>
> Key: MAPREDUCE-3372
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.23.0, 0.23.1
>Reporter: Bruno Mahé
>  Labels: bigtop
> Attachments: MAPREDUCE-3372.patch
>
>
> hadoop-config.sh forces HADOOP_prefix to a specific value:
> export HADOOP_PREFIX=`dirname "$this"`/..
> It would be nice to make this overridable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145888#comment-13145888
 ] 

Hadoop QA commented on MAPREDUCE-3342:
--

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12502820/MAPREDUCE-3342.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 9 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1259//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1259//console

This message is automatically generated.

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3372) HADOOP_PREFIX cannot be overriden

[Created]

HADOOP_PREFIX cannot be overriden
-

 Key: MAPREDUCE-3372
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3372
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé


hadoop-config.sh forces HADOOP_prefix to a specific value:
export HADOOP_PREFIX=`dirname "$this"`/..

It would be nice to make this overridable.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145883#comment-13145883
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-3342:
--



bq.  On 2011-11-07 21:59:58, Robert Evans wrote:
bq.  > 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java,
 lines 121-122
bq.  > 
bq.  >
bq.  > Could you update the comment a bit.  It kind of loses the intention 
to not fail if the format is not completely correct. Or just add in the 
try/catch blocks for the number format exception on each parse int/parselong

Going to go ahead and catch the number format exception and add a comment 
restating not to fail.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2756/#review3088
---


On 2011-11-07 22:21:11, Jonathan Eagles wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2756/
bq.  ---
bq.  
bq.  (Updated 2011-11-07 22:21:11)
bq.  
bq.  
bq.  Review request for Tom Graves, Robert Evans, Ravi Prakash, Mark 
Holderbaugh, and Anupam Seth.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The job history server doesn't show the queue the jobwas run in. It is 
inserted into the job history file.
bq.  
bq.  It seems like this should be part of the Job interface.
bq.  
bq.  JobImpl current gets it from the job config to insert into the history.
bq.  
bq.  --
bq.  Some more background. jobhistory/job jobs display a single CompletedJob 
instance which is populated by parsing the jobhistory file contents.
bq.  jobhistory/app jobs display all jobs in the history via PartialJob 
instances which are populated by parsing the jobhistory file names only, not by 
looking at the contents.
bq.  
bq.  By changing the jobhistory file naming scheme, I have disabled rigid 
parsing of job history file names and try to get as much information from them 
as we can.
bq.  
bq.  
bq.  This addresses bug MAPREDUCE-3342.
bq.  http://issues.apache.org/jira/browse/MAPREDUCE-3342
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
 07f8ecc 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
 fdba78d 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
 ea8bbdf 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
 7a6e1f0 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
 5669070 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
 f22d51c 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
 9f83b70 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
 9de3dcd 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
 71680d5 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
 fc808e5 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
 c6e3b64 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java
 6ef7bf0 
bq.  
bq.  Diff: https://reviews.apache.org/r/2756/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  In addition to unit tests. I have manually tested coexistence of old and 
new job history files, both display the queue name on the jobhistory/job page 
correctly. On the jobhistory/app page old job history file queue names will be 
displayed as null and new histor

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145882#comment-13145882
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-3342:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2756/#review3092
---

Ship it!


looks good to me +1(non-binding)

- Robert


On 2011-11-07 22:21:11, Jonathan Eagles wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2756/
bq.  ---
bq.  
bq.  (Updated 2011-11-07 22:21:11)
bq.  
bq.  
bq.  Review request for Tom Graves, Robert Evans, Ravi Prakash, Mark 
Holderbaugh, and Anupam Seth.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The job history server doesn't show the queue the jobwas run in. It is 
inserted into the job history file.
bq.  
bq.  It seems like this should be part of the Job interface.
bq.  
bq.  JobImpl current gets it from the job config to insert into the history.
bq.  
bq.  --
bq.  Some more background. jobhistory/job jobs display a single CompletedJob 
instance which is populated by parsing the jobhistory file contents.
bq.  jobhistory/app jobs display all jobs in the history via PartialJob 
instances which are populated by parsing the jobhistory file names only, not by 
looking at the contents.
bq.  
bq.  By changing the jobhistory file naming scheme, I have disabled rigid 
parsing of job history file names and try to get as much information from them 
as we can.
bq.  
bq.  
bq.  This addresses bug MAPREDUCE-3342.
bq.  http://issues.apache.org/jira/browse/MAPREDUCE-3342
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
 07f8ecc 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
 fdba78d 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
 ea8bbdf 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
 7a6e1f0 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
 5669070 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
 f22d51c 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
 9f83b70 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
 9de3dcd 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
 71680d5 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
 fc808e5 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
 c6e3b64 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java
 6ef7bf0 
bq.  
bq.  Diff: https://reviews.apache.org/r/2756/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  In addition to unit tests. I have manually tested coexistence of old and 
new job history files, both display the queue name on the jobhistory/job page 
correctly. On the jobhistory/app page old job history file queue names will be 
displayed as null and new history file queue names will be displayed correctly. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jonathan
bq.  
bq.



> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.pa

[jira] [Updated] (MAPREDUCE-3342) job history doesn't show job queue

[Jonathan Eagles (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-3342:
---

Attachment: MAPREDUCE-3342.patch

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3342) job history doesn't show job queue

[Jonathan Eagles (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-3342:
---

Status: Patch Available  (was: Open)

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch, 
> MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145878#comment-13145878
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-3342:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2756/
---

(Updated 2011-11-07 22:21:11.373425)


Review request for Tom Graves, Robert Evans, Ravi Prakash, Mark Holderbaugh, 
and Anupam Seth.


Changes
---

Addressing issue with NumberFormatException and comment


Summary
---

The job history server doesn't show the queue the jobwas run in. It is inserted 
into the job history file.

It seems like this should be part of the Job interface.

JobImpl current gets it from the job config to insert into the history.

--
Some more background. jobhistory/job jobs display a single CompletedJob 
instance which is populated by parsing the jobhistory file contents.
jobhistory/app jobs display all jobs in the history via PartialJob instances 
which are populated by parsing the jobhistory file names only, not by looking 
at the contents.

By changing the jobhistory file naming scheme, I have disabled rigid parsing of 
job history file names and try to get as much information from them as we can.


This addresses bug MAPREDUCE-3342.
http://issues.apache.org/jira/browse/MAPREDUCE-3342


Diffs (updated)
-

  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
 07f8ecc 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
 fdba78d 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
 ea8bbdf 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
 7a6e1f0 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
 5669070 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
 f22d51c 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
 9f83b70 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
 9de3dcd 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
 71680d5 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
 fc808e5 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
 c6e3b64 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java
 6ef7bf0 

Diff: https://reviews.apache.org/r/2756/diff


Testing
---

In addition to unit tests. I have manually tested coexistence of old and new 
job history files, both display the queue name on the jobhistory/job page 
correctly. On the jobhistory/app page old job history file queue names will be 
displayed as null and new history file queue names will be displayed correctly. 


Thanks,

Jonathan



> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[Jonathan Eagles (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145877#comment-13145877
 ] 

Jonathan Eagles commented on MAPREDUCE-3342:


Canceling patch while addressing comments.

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3342) job history doesn't show job queue

[Jonathan Eagles (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-3342:
---

Status: Open  (was: Patch Available)

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3291) App fail to launch due to delegation token not found in cache

[Robert Joseph Evans (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-3291:
---

Target Version/s: 0.23.0, 0.23.1  (was: 0.23.1, 0.23.0)
   Fix Version/s: 0.23.0

If you look at subversion this was merged into 0.23.0

> App fail to launch due to delegation token not found in cache
> -
>
> Key: MAPREDUCE-3291
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3291
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Ramya Sunil
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.0, 0.23.1
>
> Attachments: Log-MAPREDUCE-3291.rtf, MR-3291.txt, MR-3291.txt, 
> hadoop.err
>
>
> In secure mode, saw an app failure due to 
> "org.apache.hadoop.security.token.SecretManager$InvalidToken: token 
> (HDFS_DELEGATION_TOKEN token  for ) can't be found in cache" 
> Exception in the next comment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145862#comment-13145862
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-3342:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2756/#review3088
---



hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java


Could you update the comment a bit.  It kind of loses the intention to not 
fail if the format is not completely correct. Or just add in the try/catch 
blocks for the number format exception on each parse int/parselong


- Robert


On 2011-11-07 21:54:28, Jonathan Eagles wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/2756/
bq.  ---
bq.  
bq.  (Updated 2011-11-07 21:54:28)
bq.  
bq.  
bq.  Review request for Tom Graves, Robert Evans, Ravi Prakash, Mark 
Holderbaugh, and Anupam Seth.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  The job history server doesn't show the queue the jobwas run in. It is 
inserted into the job history file.
bq.  
bq.  It seems like this should be part of the Job interface.
bq.  
bq.  JobImpl current gets it from the job config to insert into the history.
bq.  
bq.  --
bq.  Some more background. jobhistory/job jobs display a single CompletedJob 
instance which is populated by parsing the jobhistory file contents.
bq.  jobhistory/app jobs display all jobs in the history via PartialJob 
instances which are populated by parsing the jobhistory file names only, not by 
looking at the contents.
bq.  
bq.  By changing the jobhistory file naming scheme, I have disabled rigid 
parsing of job history file names and try to get as much information from them 
as we can.
bq.  
bq.  
bq.  This addresses bug MAPREDUCE-3342.
bq.  http://issues.apache.org/jira/browse/MAPREDUCE-3342
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
 07f8ecc 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
 fdba78d 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
 ea8bbdf 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
 7a6e1f0 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
 5669070 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
 f22d51c 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
 9f83b70 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
 9de3dcd 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
 71680d5 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
 fc808e5 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
 c6e3b64 
bq.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java
 6ef7bf0 
bq.  
bq.  Diff: https://reviews.apache.org/r/2756/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  In addition to unit tests. I have manually tested coexistence of old and 
new job history files, both display the queue name on the jobhistory/job page 
correctly. On the jobhistory/app page old job history file queue names will be 
displayed as null and new history file queue names will be displayed correctly. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jonathan
bq.  
bq.



> job history doesn't show job queue
> --
>
> 

[jira] [Updated] (MAPREDUCE-3342) job history doesn't show job queue

[Jonathan Eagles (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-3342:
---

Status: Patch Available  (was: Open)

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145854#comment-13145854
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-3342:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2756/
---

(Updated 2011-11-07 21:54:28.706876)


Review request for Tom Graves, Robert Evans, Ravi Prakash, Mark Holderbaugh, 
and Anupam Seth.


Changes
---

Fix TestHSWebApp test


Summary
---

The job history server doesn't show the queue the jobwas run in. It is inserted 
into the job history file.

It seems like this should be part of the Job interface.

JobImpl current gets it from the job config to insert into the history.

--
Some more background. jobhistory/job jobs display a single CompletedJob 
instance which is populated by parsing the jobhistory file contents.
jobhistory/app jobs display all jobs in the history via PartialJob instances 
which are populated by parsing the jobhistory file names only, not by looking 
at the contents.

By changing the jobhistory file naming scheme, I have disabled rigid parsing of 
job history file names and try to get as much information from them as we can.


This addresses bug MAPREDUCE-3342.
http://issues.apache.org/jira/browse/MAPREDUCE-3342


Diffs (updated)
-

  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
 07f8ecc 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
 fdba78d 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
 ea8bbdf 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
 7a6e1f0 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
 5669070 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
 f22d51c 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
 9f83b70 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
 9de3dcd 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
 71680d5 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
 fc808e5 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
 c6e3b64 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java
 6ef7bf0 

Diff: https://reviews.apache.org/r/2756/diff


Testing
---

In addition to unit tests. I have manually tested coexistence of old and new 
job history files, both display the queue name on the jobhistory/job page 
correctly. On the jobhistory/app page old job history file queue names will be 
displayed as null and new history file queue names will be displayed correctly. 


Thanks,

Jonathan



> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3342) job history doesn't show job queue

[Jonathan Eagles (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-3342:
---

Attachment: MAPREDUCE-3342.patch

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch, MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3342) job history doesn't show job queue

[Jonathan Eagles (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-3342:
---

Status: Open  (was: Patch Available)

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145838#comment-13145838
 ] 

Hadoop QA commented on MAPREDUCE-3342:
--

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12502809/MAPREDUCE-3342.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 9 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

-1 core tests.  The patch failed these unit tests:
  org.apache.hadoop.mapreduce.v2.hs.webapp.TestHSWebApp

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1258//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1258//console

This message is automatically generated.

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3342) job history doesn't show job queue

[jirapos...@reviews.apache.org (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145833#comment-13145833
 ] 

jirapos...@reviews.apache.org commented on MAPREDUCE-3342:
--


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2756/
---

Review request for Tom Graves, Robert Evans, Ravi Prakash, Mark Holderbaugh, 
and Anupam Seth.


Summary
---

The job history server doesn't show the queue the jobwas run in. It is inserted 
into the job history file.

It seems like this should be part of the Job interface.

JobImpl current gets it from the job config to insert into the history.

--
Some more background. jobhistory/job jobs display a single CompletedJob 
instance which is populated by parsing the jobhistory file contents.
jobhistory/app jobs display all jobs in the history via PartialJob instances 
which are populated by parsing the jobhistory file names only, not by looking 
at the contents.

By changing the jobhistory file naming scheme, I have disabled rigid parsing of 
job history file names and try to get as much information from them as we can.


This addresses bug MAPREDUCE-3342.
http://issues.apache.org/jira/browse/MAPREDUCE-3342


Diffs
-

  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
 07f8ecc 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
 fdba78d 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
 ea8bbdf 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
 7a6e1f0 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
 5669070 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java
 f22d51c 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java
 9f83b70 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java
 9de3dcd 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
 71680d5 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
 fc808e5 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java
 c6e3b64 
  
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java
 6ef7bf0 

Diff: https://reviews.apache.org/r/2756/diff


Testing
---

In addition to unit tests. I have manually tested coexistence of old and new 
job history files, both display the queue name on the jobhistory/job page 
correctly. On the jobhistory/app page old job history file queue names will be 
displayed as null and new history file queue names will be displayed correctly. 


Thanks,

Jonathan



> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3342) job history doesn't show job queue

[Jonathan Eagles (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-3342:
---

Attachment: MAPREDUCE-3342.patch

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3342) job history doesn't show job queue

[Jonathan Eagles (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Eagles updated MAPREDUCE-3342:
---

Status: Patch Available  (was: Open)

> job history doesn't show job queue
> --
>
> Key: MAPREDUCE-3342
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3342
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Jonathan Eagles
>Priority: Critical
> Attachments: MAPREDUCE-3342.patch
>
>
> The job history server doesn't show the queue the jobwas run in.  It is 
> inserted into the job history file.
> It seems like this should be part of the Job interface.
> JobImpl current gets it from the job config to insert into the history.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3326) RM web UI scheduler link not as useful as should be

[Jonathan Eagles (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145810#comment-13145810
 ] 

Jonathan Eagles commented on MAPREDUCE-3326:


Not missing anything. 20 and 22 will be a better reference for what this should 
look like with the addition MAPREDUCE-1118.

> RM web UI scheduler link not as useful as should be
> ---
>
> Key: MAPREDUCE-3326
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3326
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Anupam Seth
>Priority: Critical
>
> The resource manager web ui page for scheduler doesn't have all the 
> information about the configuration like the jobtracker page used to have.  
> The things it seems to show you are the current queues - each queues used, 
> set, and max percent and then what apps are running in that queue.  
> It doesn't list any of yarn.scheduler.capacity.maximum-applications, 
> yarn.scheduler.capacity.maximum-am-resource-percent, 
> yarn.scheduler.capacity..user-limit-factor, 
> yarn.scheduler.capacity..minimum-user-limit-percent, queue state, 
> active users and percent used by user 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3371) Review and improve the yarn-api javadocs.

[Ravi Prakash (Created) (JIRA)]

Review and improve the yarn-api javadocs.
-

 Key: MAPREDUCE-3371
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3371
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: documentation, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Priority: Minor
 Fix For: 0.23.1


Review and improve the yarn-api javadocs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3326) RM web UI scheduler link not as useful as should be

[Anupam Seth (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145805#comment-13145805
 ] 

Anupam Seth commented on MAPREDUCE-3326:


Thanks Jon! 

Not sure if I am missing / misunderstanding something, but upon reading the 
discussion, it seems that MAPREDUCE-1118 applies only to 0.20 and 0.22 and not 
0.23 / mrv2.



> RM web UI scheduler link not as useful as should be
> ---
>
> Key: MAPREDUCE-3326
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3326
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Anupam Seth
>Priority: Critical
>
> The resource manager web ui page for scheduler doesn't have all the 
> information about the configuration like the jobtracker page used to have.  
> The things it seems to show you are the current queues - each queues used, 
> set, and max percent and then what apps are running in that queue.  
> It doesn't list any of yarn.scheduler.capacity.maximum-applications, 
> yarn.scheduler.capacity.maximum-am-resource-percent, 
> yarn.scheduler.capacity..user-limit-factor, 
> yarn.scheduler.capacity..minimum-user-limit-percent, queue state, 
> active users and percent used by user 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3344) o.a.h.mapreduce.Reducer since 0.21 blindly casts to ReduceContext.ValueIterator

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145804#comment-13145804
 ] 

Hudson commented on MAPREDUCE-3344:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #1274 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1274/])
MAPREDUCE-3344. o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
ReduceContext.ValueIterator. Contributed by Brock Noland.

tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198910
Files : 
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java


> o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
> ReduceContext.ValueIterator
> ---
>
> Key: MAPREDUCE-3344
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3344
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.23.1
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3344-1.patch
>
>
> 0.21 mapreduce.Reducer introduced a blind cast to 
> ReduceContext.ValueIterator. There should an instanceof check around this 
> block to ensure we don't throw a CastClassException:
> {code}
>// If a back up store is used, reset it
>   ((ReduceContext.ValueIterator)
>   (context.getValues().iterator())).resetBackupStore();
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2

[Ahmed Radwan (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145802#comment-13145802
 ] 

Ahmed Radwan commented on MAPREDUCE-3169:
-

For organization purpose, please note that the actual changes made to specific 
tests for the purpose of migration are tacked in MAPREDUCE-3369. This current 
ticket is only tracking general interfaces changes.

> Create a new MiniMRCluster equivalent which only provides client APIs cross 
> MR1 and MR2
> ---
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2, test
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, 
> MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster 
> for writing tests. Many users do as well. MiniMRCluster, however, exposes MR 
> implementation details like the existence of TaskTrackers, JobTrackers, etc, 
> since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by 
> MR1 or MR2 that exposes only the client-side portions of the MR framework. 
> Ideally it would be "recompile-compatible" with MiniMRCluster for most 
> applications, and the MR1 implementation could be backported to 20x branch. 
> Thus, dependent projects like HBase could migrate to this implementation and 
> test against both MR1 and MR2. We can also use this to port over the current 
> functional tests that use only the client-side features of MiniMRCluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3169) Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2

[Ahmed Radwan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-3169:


Attachment: MAPREDUCE-3169-trunk_rev2.patch

A revised patch for trunk. The patch mainly adds a mock class MiniMRCluster 
that wraps the new interfaces, so old tests can run with minimal changes. There 
are also some minor changes to the patch based on issues faced during tests 
migration.

> Create a new MiniMRCluster equivalent which only provides client APIs cross 
> MR1 and MR2
> ---
>
> Key: MAPREDUCE-3169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3169
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2, test
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3169-0.20-security.patch, 
> MAPREDUCE-3169-truck.patch, MAPREDUCE-3169-trunk_rev2.patch
>
>
> Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster 
> for writing tests. Many users do as well. MiniMRCluster, however, exposes MR 
> implementation details like the existence of TaskTrackers, JobTrackers, etc, 
> since it was used by MR1 for testing the server implementations as well.
> This JIRA is to create a new interface which could be implemented either by 
> MR1 or MR2 that exposes only the client-side portions of the MR framework. 
> Ideally it would be "recompile-compatible" with MiniMRCluster for most 
> applications, and the MR1 implementation could be backported to 20x branch. 
> Thus, dependent projects like HBase could migrate to this implementation and 
> test against both MR1 and MR2. We can also use this to port over the current 
> functional tests that use only the client-side features of MiniMRCluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3369) Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169

[Ahmed Radwan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-3369:


Attachment: MAPREDUCE-3369.patch

The first round of tests migration. Required making some changes to some tests, 
in addition to some pom.xml project dependencies to pick moved files. Most of 
test classes in the patch pass successfully, few have failures, which may be 
bugs in MR2 or further changes to be done in the tests.

To run the tests, please apply first the patch from MAPREDUCE-3370.

> Migrate MR1 tests to run on MR2 using the new interfaces introduced in 
> MAPREDUCE-3169
> -
>
> Key: MAPREDUCE-3369
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3369
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mrv1, mrv2, test
>Affects Versions: 0.23.0
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3369.patch
>
>
> This ticket tracks the migration of MR1 tests (currently residing in 
> "hadoop-mapreduce-project/src/test/") to run on MR2. The migration is using 
> the new interfaces introduced in MAPREDUCE-3169.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3344) o.a.h.mapreduce.Reducer since 0.21 blindly casts to ReduceContext.ValueIterator

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145789#comment-13145789
 ] 

Hudson commented on MAPREDUCE-3344:
---

Integrated in Hadoop-Mapreduce-0.23-Commit #163 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/163/])
Merge -r 1198909:1198910 from trunk to branch-0.23. Fixes: MAPREDUCE-3344.

tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198914
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java


> o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
> ReduceContext.ValueIterator
> ---
>
> Key: MAPREDUCE-3344
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3344
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.23.1
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3344-1.patch
>
>
> 0.21 mapreduce.Reducer introduced a blind cast to 
> ReduceContext.ValueIterator. There should an instanceof check around this 
> block to ensure we don't throw a CastClassException:
> {code}
>// If a back up store is used, reset it
>   ((ReduceContext.ValueIterator)
>   (context.getValues().iterator())).resetBackupStore();
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3370) MiniMRYarnCluster uses a hard coded path location for the MapReduce application jar

[Ahmed Radwan (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-3370:


Attachment: MAPREDUCE-3370.patch

The patch uses a system property for MR application jar location.

> MiniMRYarnCluster uses a hard coded path location for the MapReduce 
> application jar
> ---
>
> Key: MAPREDUCE-3370
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3370
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2, test
>Affects Versions: 0.23.0
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-3370.patch
>
>
> MiniMRYarnCluster uses a hard coded relative path location for the MapReduce 
> application jar. It is better to have this location as a system property so 
> tests can pick the application jar regardless of their working directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3370) MiniMRYarnCluster uses a hard coded path location for the MapReduce application jar

[Ahmed Radwan (Created) (JIRA)]

MiniMRYarnCluster uses a hard coded path location for the MapReduce application 
jar
---

 Key: MAPREDUCE-3370
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3370
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 0.23.0
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan


MiniMRYarnCluster uses a hard coded relative path location for the MapReduce 
application jar. It is better to have this location as a system property so 
tests can pick the application jar regardless of their working directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3326) RM web UI scheduler link not as useful as should be

[Jonathan Eagles (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145783#comment-13145783
 ] 

Jonathan Eagles commented on MAPREDUCE-3326:


A recent jira MAPREDUCE-1118 may have updated the displayed capacity scheduler 
information and certainly has changed the layout of the information.

> RM web UI scheduler link not as useful as should be
> ---
>
> Key: MAPREDUCE-3326
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3326
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Anupam Seth
>Priority: Critical
>
> The resource manager web ui page for scheduler doesn't have all the 
> information about the configuration like the jobtracker page used to have.  
> The things it seems to show you are the current queues - each queues used, 
> set, and max percent and then what apps are running in that queue.  
> It doesn't list any of yarn.scheduler.capacity.maximum-applications, 
> yarn.scheduler.capacity.maximum-am-resource-percent, 
> yarn.scheduler.capacity..user-limit-factor, 
> yarn.scheduler.capacity..minimum-user-limit-percent, queue state, 
> active users and percent used by user 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3369) Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169

[Ahmed Radwan (Created) (JIRA)]

Migrate MR1 tests to run on MR2 using the new interfaces introduced in 
MAPREDUCE-3169
-

 Key: MAPREDUCE-3369
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3369
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1, mrv2, test
Affects Versions: 0.23.0
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan


This ticket tracks the migration of MR1 tests (currently residing in 
"hadoop-mapreduce-project/src/test/") to run on MR2. The migration is using the 
new interfaces introduced in MAPREDUCE-3169.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-3326) RM web UI scheduler link not as useful as should be

[Anupam Seth (Assigned) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anupam Seth reassigned MAPREDUCE-3326:
--

Assignee: Anupam Seth

> RM web UI scheduler link not as useful as should be
> ---
>
> Key: MAPREDUCE-3326
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3326
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Thomas Graves
>Assignee: Anupam Seth
>Priority: Critical
>
> The resource manager web ui page for scheduler doesn't have all the 
> information about the configuration like the jobtracker page used to have.  
> The things it seems to show you are the current queues - each queues used, 
> set, and max percent and then what apps are running in that queue.  
> It doesn't list any of yarn.scheduler.capacity.maximum-applications, 
> yarn.scheduler.capacity.maximum-am-resource-percent, 
> yarn.scheduler.capacity..user-limit-factor, 
> yarn.scheduler.capacity..minimum-user-limit-percent, queue state, 
> active users and percent used by user 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3344) o.a.h.mapreduce.Reducer since 0.21 blindly casts to ReduceContext.ValueIterator

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145772#comment-13145772
 ] 

Hudson commented on MAPREDUCE-3344:
---

Integrated in Hadoop-Common-0.23-Commit #152 (See 
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/152/])
Merge -r 1198909:1198910 from trunk to branch-0.23. Fixes: MAPREDUCE-3344.

tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198914
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java


> o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
> ReduceContext.ValueIterator
> ---
>
> Key: MAPREDUCE-3344
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3344
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.23.1
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3344-1.patch
>
>
> 0.21 mapreduce.Reducer introduced a blind cast to 
> ReduceContext.ValueIterator. There should an instanceof check around this 
> block to ensure we don't throw a CastClassException:
> {code}
>// If a back up store is used, reset it
>   ((ReduceContext.ValueIterator)
>   (context.getValues().iterator())).resetBackupStore();
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3344) o.a.h.mapreduce.Reducer since 0.21 blindly casts to ReduceContext.ValueIterator

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145769#comment-13145769
 ] 

Hudson commented on MAPREDUCE-3344:
---

Integrated in Hadoop-Common-trunk-Commit #1252 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1252/])
MAPREDUCE-3344. o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
ReduceContext.ValueIterator. Contributed by Brock Noland.

tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198910
Files : 
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java


> o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
> ReduceContext.ValueIterator
> ---
>
> Key: MAPREDUCE-3344
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3344
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.23.1
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3344-1.patch
>
>
> 0.21 mapreduce.Reducer introduced a blind cast to 
> ReduceContext.ValueIterator. There should an instanceof check around this 
> block to ensure we don't throw a CastClassException:
> {code}
>// If a back up store is used, reset it
>   ((ReduceContext.ValueIterator)
>   (context.getValues().iterator())).resetBackupStore();
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3344) o.a.h.mapreduce.Reducer since 0.21 blindly casts to ReduceContext.ValueIterator

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145767#comment-13145767
 ] 

Hudson commented on MAPREDUCE-3344:
---

Integrated in Hadoop-Hdfs-trunk-Commit #1326 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1326/])
MAPREDUCE-3344. o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
ReduceContext.ValueIterator. Contributed by Brock Noland.

tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198910
Files : 
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java


> o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
> ReduceContext.ValueIterator
> ---
>
> Key: MAPREDUCE-3344
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3344
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.23.1
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3344-1.patch
>
>
> 0.21 mapreduce.Reducer introduced a blind cast to 
> ReduceContext.ValueIterator. There should an instanceof check around this 
> block to ensure we don't throw a CastClassException:
> {code}
>// If a back up store is used, reset it
>   ((ReduceContext.ValueIterator)
>   (context.getValues().iterator())).resetBackupStore();
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3344) o.a.h.mapreduce.Reducer since 0.21 blindly casts to ReduceContext.ValueIterator

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145768#comment-13145768
 ] 

Hudson commented on MAPREDUCE-3344:
---

Integrated in Hadoop-Hdfs-0.23-Commit #151 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/151/])
Merge -r 1198909:1198910 from trunk to branch-0.23. Fixes: MAPREDUCE-3344.

tomwhite : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1198914
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java


> o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
> ReduceContext.ValueIterator
> ---
>
> Key: MAPREDUCE-3344
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3344
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.23.1
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3344-1.patch
>
>
> 0.21 mapreduce.Reducer introduced a blind cast to 
> ReduceContext.ValueIterator. There should an instanceof check around this 
> block to ensure we don't throw a CastClassException:
> {code}
>// If a back up store is used, reset it
>   ((ReduceContext.ValueIterator)
>   (context.getValues().iterator())).resetBackupStore();
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3344) o.a.h.mapreduce.Reducer since 0.21 blindly casts to ReduceContext.ValueIterator

[Tom White (Updated) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom White updated MAPREDUCE-3344:
-

  Resolution: Fixed
   Fix Version/s: 0.23.1
  0.24.0
Assignee: Brock Noland
Target Version/s: 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.23.1  (was: 0.23.1, 
0.24.0, 0.23.0, 0.22.0, 0.21.0)
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

I've just committed this. Thanks, Brock!

> o.a.h.mapreduce.Reducer since 0.21 blindly casts to 
> ReduceContext.ValueIterator
> ---
>
> Key: MAPREDUCE-3344
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3344
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.23.1
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.24.0, 0.23.1
>
> Attachments: MAPREDUCE-3344-1.patch
>
>
> 0.21 mapreduce.Reducer introduced a blind cast to 
> ReduceContext.ValueIterator. There should an instanceof check around this 
> block to ensure we don't throw a CastClassException:
> {code}
>// If a back up store is used, reset it
>   ((ReduceContext.ValueIterator)
>   (context.getValues().iterator())).resetBackupStore();
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3323) Add new interface for Distributed Cache, which special for Map or Reduce,but not Both.

[Tom White (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145721#comment-13145721
 ] 

Tom White commented on MAPREDUCE-3323:
--

DistributedCache is deprecated in 0.23 and trunk in favour of methods on Job, 
so please make sure that you add equivalent methods to Job. It might be a good 
idea to use TaskType as an overloaded parameter, e.g.

{noformat}
public void addCacheFile(URI uri)
public void addCacheFile(URI uri, TaskType taskType)
{noformat}


> Add new interface for Distributed Cache, which special  for Map or Reduce,but 
> not Both.
> ---
>
> Key: MAPREDUCE-3323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3323
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: distributed-cache, tasktracker
>Affects Versions: 0.20.203.0
>Reporter: Azuryy(Chijiong)
> Fix For: 0.20.203.0
>
> Attachments: DistributedCache.patch, GenericOptionsParser.patch, 
> JobClient.patch, TaskDistributedCacheManager.patch, TaskTracker.patch
>
>
> We put some file into Distributed Cache, but sometimes, only Map or Reduce 
> use thses cached files, not useful for both. but TaskTracker always download 
> cached files from HDFS, if there are some little bit big files in cache, it's 
> time expensive.
> so, this patch add some new API in the DistributedCache.java as follow:
> addArchiveToClassPathForMap
> addArchiveToClassPathForReduce
> addFileToClassPathForMap
> addFileToClassPathForReduce
> addCacheFileForMap
> addCacheFileForReduce
> addCacheArchiveForMap
> addCacheArchiveForReduce
> New API doesn't affect original interface. User can use these features like 
> the following two methods:
> 1) 
> hadoop job  -files file1 -mapfiles file2 -reducefiles file3 -archives 
> arc1 -maparchives arc2 -reduce archives arc3
> 2)
> DistributedCache.addCacheFile(conf, file1);
> DistributedCache.addCacheFileForMap(conf, file2);
> DistributedCache.addCacheFileForReduce(conf, file3);
> DistributedCache.addCacheArchives(conf, arc1);
> DistributedCache.addCacheArchivesForMap(conf, arc2);
> DistributedCache.addCacheFArchivesForReduce(conf, arc3);
> These two methods have the same result, That's mean: 
> You put six files to the distributed cache: file1 ~ file3, arc1 ~ arc3, 
> but file1 and arc1 are cached for both map and reduce;
> file2 and arc2 are only cached for map;
> file3 and arc3 are only cached for reduce;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3368) compile-mapred-test fails

[Ramya Sunil (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145698#comment-13145698
 ] 

Ramya Sunil commented on MAPREDUCE-3368:


{code}
compile-mapred-test:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/build/test/mapred/classes
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/build/test/mapred/testjar
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/build/test/mapred/testshell
[javac] Compiling 332 source files to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/build/test/mapred/classes
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestAuditLogger.java:24:
 cannot find symbol
[javac] symbol  : class ProtocolInfo
[javac] location: package org.apache.hadoop.ipc
[javac] import org.apache.hadoop.ipc.ProtocolInfo;
[javac] ^
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestAuditLogger.java:127:
 cannot find symbol
[javac] symbol  : class ProtocolInfo
[javac] location: class org.apache.hadoop.mapred.TestAuditLogger
[javac]   @ProtocolInfo(protocolName = 
"org.apache.hadoop.ipc.TestRPC$TestProtocol")
[javac]^
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:36:
 package org.apache.hadoop.hdfs.protocolR23Compatible does not exist
[javac] import 
org.apache.hadoop.hdfs.protocolR23Compatible.ClientNamenodeWireProtocol;
[javac]^
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:151:
 cannot find symbol
[javac] symbol  : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac]   static ClientNamenodeWireProtocol getDFSClient(
[javac]  ^
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java:105:
 warning: [unchecked] unchecked conversion
[javac] found   : org.apache.hadoop.mapred.IFile.Writer
[javac] required: 
org.apache.hadoop.mapred.IFile.Writer
[javac] Writer mockWriter = mock(Writer.class);
[javac]  ^
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java:125:
 warning: [unchecked] unchecked conversion
[javac] found   : org.apache.hadoop.mapred.IFile.Writer
[javac] required: 
org.apache.hadoop.mapred.IFile.Writer
[javac] Writer mockWriter = mock(Writer.class);
[javac]  ^
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:154:
 cannot find symbol
[javac] symbol  : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac] return (ClientNamenodeWireProtocol) 
[javac] ^
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
 cannot find symbol
[javac] symbol  : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac]   RPC.getProxy(ClientNamenodeWireProtocol.class, 
[javac]^
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:156:
 cannot find symbol
[javac] symbol  : variable ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac]   ClientNamenodeWireProtocol.versionID, 
[javac]   ^
[javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:160:
 cannot find symbol
[javac] symbol  : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac]

[jira] [Created] (MAPREDUCE-3368) compile-mapred-test fails

[Ramya Sunil (Created) (JIRA)]

compile-mapred-test fails
-

 Key: MAPREDUCE-3368
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3368
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: build, mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Priority: Critical
 Fix For: 0.23.0


compile-mapred-test target is failing once again.
Details: 
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-0.23-Build/83/consoleFull

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira