[jira] [Commented] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398230#comment-13398230
]
Devaraj K commented on MAPREDUCE-4355:
--
It is good to have this API. Thanks for providing patch.
There are few minor comments about the patch
1. Can we get the JobStatus directly from Cluster instance instead of getting
the Job instance and getting the JobStatus from Job instance? In Cluster.java,
getJob(JobID jobId) API is getting the JobStatus and creating the Job instance
using the JobStatus, again you are getting the JobStatus from Job instance.
2. I don't see any use of having the jobid as final.
{code:xml}
+ public JobStatus getJobStatus(final JobID jobid) throws IOException {
{code}
3. Can you rename the jobid variable name to jobId, anyway it is minor you can
take the final decision.
> Add JobStatus getJobStatus(JobID) to JobClient.
> ---
>
> Key: MAPREDUCE-4355
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv1, mrv2
>Affects Versions: 1.0.3, 2.0.0-alpha
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch
>
>
> To read the start-time of a particular job, one should not need to
> getAllJobs() and iterate through them.
> getJob(JobID) returns RunningJob, which doesn't hold the job's start time.
> Hence, we need to add getJobStatus(JobID) to the API.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4356) Provide access to ParsedTask.obtainTaskAttempts()
[ https://issues.apache.org/jira/browse/MAPREDUCE-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398219#comment-13398219 ] Hudson commented on MAPREDUCE-4356: --- Integrated in Hadoop-Common-trunk-Commit #2375 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2375/]) MAPREDUCE-4356. [Rumen] Provide access to the method ParsedTask.obtainTaskAttempts(). (ravigummadi) (Revision 1352415) Result = SUCCESS ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352415 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ParsedTask.java > Provide access to ParsedTask.obtainTaskAttempts() > - > > Key: MAPREDUCE-4356 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4356 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tools/rumen >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4356.patch > > > Change the access modifier of obtainTaskAttempts() in ParsedTask.java from > default to public sothat it is accessible for everyone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4356) Provide access to ParsedTask.obtainTaskAttempts()
[ https://issues.apache.org/jira/browse/MAPREDUCE-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398217#comment-13398217 ] Hudson commented on MAPREDUCE-4356: --- Integrated in Hadoop-Hdfs-trunk-Commit #2445 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2445/]) MAPREDUCE-4356. [Rumen] Provide access to the method ParsedTask.obtainTaskAttempts(). (ravigummadi) (Revision 1352415) Result = SUCCESS ravigummadi : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352415 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/ParsedTask.java > Provide access to ParsedTask.obtainTaskAttempts() > - > > Key: MAPREDUCE-4356 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4356 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tools/rumen >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4356.patch > > > Change the access modifier of obtainTaskAttempts() in ParsedTask.java from > default to public sothat it is accessible for everyone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4356) Provide access to ParsedTask.obtainTaskAttempts()
[ https://issues.apache.org/jira/browse/MAPREDUCE-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-4356: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this to trunk. > Provide access to ParsedTask.obtainTaskAttempts() > - > > Key: MAPREDUCE-4356 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4356 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tools/rumen >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4356.patch > > > Change the access modifier of obtainTaskAttempts() in ParsedTask.java from > default to public sothat it is accessible for everyone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4290) JobStatus.getState() API is giving ambiguous values
[ https://issues.apache.org/jira/browse/MAPREDUCE-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-4290: - Status: Patch Available (was: Open) > JobStatus.getState() API is giving ambiguous values > --- > > Key: MAPREDUCE-4290 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4290 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Nishan Shetty >Assignee: Devaraj K > Attachments: MAPREDUCE-4290-1.patch, MAPREDUCE-4290.patch > > > For failed job getState() API is giving status as SUCCEEDED if we use > JobClient.getAllJobs() for retrieving all jobs info from RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4290) JobStatus.getState() API is giving ambiguous values
[ https://issues.apache.org/jira/browse/MAPREDUCE-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-4290: - Status: Open (was: Patch Available) > JobStatus.getState() API is giving ambiguous values > --- > > Key: MAPREDUCE-4290 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4290 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Nishan Shetty >Assignee: Devaraj K > Attachments: MAPREDUCE-4290-1.patch, MAPREDUCE-4290.patch > > > For failed job getState() API is giving status as SUCCEEDED if we use > JobClient.getAllJobs() for retrieving all jobs info from RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4356) Provide access to ParsedTask.obtainTaskAttempts()
[ https://issues.apache.org/jira/browse/MAPREDUCE-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398199#comment-13398199 ] Amar Kamat commented on MAPREDUCE-4356: --- +1. > Provide access to ParsedTask.obtainTaskAttempts() > - > > Key: MAPREDUCE-4356 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4356 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tools/rumen >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 4356.patch > > > Change the access modifier of obtainTaskAttempts() in ParsedTask.java from > default to public sothat it is accessible for everyone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3868) Reenable Raid
[ https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398178#comment-13398178 ] Scott Chen commented on MAPREDUCE-3868: --- Sorry for causing the trouble. I should have run findbugs before checking this in. > Reenable Raid > - > > Key: MAPREDUCE-3868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Reporter: Scott Chen >Assignee: Weiyan Wang > Attachments: MAPREDUCE-3868-1.patch, MAPREDUCE-3868-2.patch, > MAPREDUCE-3868-3.patch, MAPREDUCE-3868.patch, MAPREDUCE-3868v1.patch, > MAPREDUCE-3868v1.sh > > > Currently Raid is outdated and not compiled. Make it compile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398164#comment-13398164 ] Mayank Bansal commented on MAPREDUCE-3837: -- When I put this patch it did not have this issue,Let me update the patch. Thanks for finding this out. Thanks, Mayank > Hadoop 22 Job tracker is not able to recover job in case of crash and after > that no user can submit job. > > > Key: MAPREDUCE-3837 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Fix For: 0.24.0, 0.22.1, 0.23.2 > > Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, > PATCH-HADOOP-1-MAPREDUCE-3837-2.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, > PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch > > > If job tracker is crashed while running , and there were some jobs are > running , so if job tracker's property mapreduce.jobtracker.restart.recover > is true then it should recover the job. > However the current behavior is as follows > jobtracker try to restore the jobs but it can not . And after that jobtracker > closes its handle to hdfs and nobody else can submit job. > Thanks, > Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4203) Create equivalent of ProcfsBasedProcessTree for Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-4203. -- Resolution: Fixed Fix Version/s: 1-win Release Note: Added an implementation of the process tree for Windows. I just committed this. Thanks Bikas! > Create equivalent of ProcfsBasedProcessTree for Windows > --- > > Key: MAPREDUCE-4203 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 1-win > > Attachments: MAPREDUCE-4203.branch-1-win.1.patch, > MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, > MAPREDUCE-4203.branch-1-win.4.patch, MAPREDUCE-4203.patch, test.cpp > > > ProcfsBasedProcessTree is used by the TaskTracker to get process information > like memory and cpu usage. This information is used to manage resources etc. > The current implementation is based on Linux procfs functionality and hence > does not work on other platforms, specifically windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397998#comment-13397998 ] Jason Lowe commented on MAPREDUCE-4355: --- The 13 javadoc warnings were caused by MAPREDUCE-3868, see HDFS-3550. > Add JobStatus getJobStatus(JobID) to JobClient. > --- > > Key: MAPREDUCE-4355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv1, mrv2 >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch > > > To read the start-time of a particular job, one should not need to > getAllJobs() and iterate through them. > getJob(JobID) returns RunningJob, which doesn't hold the job's start time. > Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3868) Reenable Raid
[ https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397974#comment-13397974 ] Jason Lowe commented on MAPREDUCE-3868: --- This also broke the dist tar build - see HDFS-3549. > Reenable Raid > - > > Key: MAPREDUCE-3868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Reporter: Scott Chen >Assignee: Weiyan Wang > Attachments: MAPREDUCE-3868-1.patch, MAPREDUCE-3868-2.patch, > MAPREDUCE-3868-3.patch, MAPREDUCE-3868.patch, MAPREDUCE-3868v1.patch, > MAPREDUCE-3868v1.sh > > > Currently Raid is outdated and not compiled. Make it compile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397964#comment-13397964 ] Andrew Ferguson commented on MAPREDUCE-4351: Another libstdc++ failure from pipes. > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch, > MAPREDUCE-4351-v5.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397960#comment-13397960 ] Hudson commented on MAPREDUCE-3889: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2393 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2393/]) MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java > job client tries to use /tasklog interface, but that doesn't exist anymore > -- > > Key: MAPREDUCE-3889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 2.0.1-alpha, 3.0.0 >Reporter: Thomas Graves >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 > > Attachments: MAPREDUCE-3889.patch, MAPREDUCE-3889.patch > > > if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when > running a job it tries to fetch task logs to print out on the client side > from a url like: > http://nodemanager:8080/tasklog?plaintext=true&attemptid=attempt_1329857083014_0003_r_00_0&filter=stdout > It always errors on this request with: Required param job, map and reduce > We saw this error when using distcp and the distcp failed. I'm not sure if it > is mandatory for distcp or just informational purposes. I'm guessing the > latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397945#comment-13397945 ] Karthik Kambatla commented on MAPREDUCE-4355: - W.r.t MR2 patch, Eclipse shows the same number of warnings with and without the patch. Any pointers as to where I should look for these extra warnings Hudson reports? Otherwise, I believe the patches are good to go. > Add JobStatus getJobStatus(JobID) to JobClient. > --- > > Key: MAPREDUCE-4355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv1, mrv2 >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch > > > To read the start-time of a particular job, one should not need to > getAllJobs() and iterate through them. > getJob(JobID) returns RunningJob, which doesn't hold the job's start time. > Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397939#comment-13397939 ] Hadoop QA commented on MAPREDUCE-4355: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532781/MR-4355_mr1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2489//console This message is automatically generated. > Add JobStatus getJobStatus(JobID) to JobClient. > --- > > Key: MAPREDUCE-4355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv1, mrv2 >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch > > > To read the start-time of a particular job, one should not need to > getAllJobs() and iterate through them. > getJob(JobID) returns RunningJob, which doesn't hold the job's start time. > Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3642) Remove hardcoded strings from the JC#displayTasks() call.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397935#comment-13397935 ] Tsuyoshi OZAWA commented on MAPREDUCE-3642: --- Harsh, I'd like to deal with this ticket, however, I don't have a contributor privilege. Could you give it to me? > Remove hardcoded strings from the JC#displayTasks() call. > - > > Key: MAPREDUCE-3642 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3642 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 0.24.0 >Reporter: Harsh J > Labels: newbie > Attachments: MR-3642.1.patch > > > This is to address Eli's comments on the parent task: > bq. 1. The error messages should generate the lists of valid states and types > from their definitions rather than hard-coding them into the error messages. > bq. 2. Aren't these types and states defined somewhere already? Seems like > they're a public API and therefore shouldn't have to duplicate the definition > of them in taskTypes and taskStates. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4355: Attachment: MR-4355_mr1.patch > Add JobStatus getJobStatus(JobID) to JobClient. > --- > > Key: MAPREDUCE-4355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv1, mrv2 >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch > > > To read the start-time of a particular job, one should not need to > getAllJobs() and iterate through them. > getJob(JobID) returns RunningJob, which doesn't hold the job's start time. > Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397920#comment-13397920 ] Hadoop QA commented on MAPREDUCE-4351: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532778/MAPREDUCE-4351-v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2488//console This message is automatically generated. > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch, > MAPREDUCE-4351-v5.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4266) remove Ant remnants from MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397915#comment-13397915 ] Alejandro Abdelnur commented on MAPREDUCE-4266: --- Oops, ok go it, yes a script would be more convenient I'd say. > remove Ant remnants from MR > --- > > Key: MAPREDUCE-4266 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4266 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 2.0.1-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.1-alpha > > Attachments: MAPREDUCE-4266.patch > > > Remove: > hadoop-mapreduce-project/src/* > hadoop-mapreduce-project/ivy/* > hadoop-mapreduce-project/build.xml > hadoop-mapreduce-project/ivy.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3889: --- Resolution: Fixed Fix Version/s: 3.0.0 2.0.1-alpha 0.23.3 Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.0-alpha, 3.0.0) Status: Resolved (was: Patch Available) Thanks Devaraj, Keep up the good work. I put this into branch-0.23, branch-2, and trunk. > job client tries to use /tasklog interface, but that doesn't exist anymore > -- > > Key: MAPREDUCE-3889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 2.0.1-alpha, 3.0.0 >Reporter: Thomas Graves >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 > > Attachments: MAPREDUCE-3889.patch, MAPREDUCE-3889.patch > > > if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when > running a job it tries to fetch task logs to print out on the client side > from a url like: > http://nodemanager:8080/tasklog?plaintext=true&attemptid=attempt_1329857083014_0003_r_00_0&filter=stdout > It always errors on this request with: Required param job, map and reduce > We saw this error when using distcp and the distcp failed. I'm not sure if it > is mandatory for distcp or just informational purposes. I'm guessing the > latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Ferguson updated MAPREDUCE-4351: --- Attachment: MAPREDUCE-4351-v5.patch Add documentation to ContainersMonitor > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch, > MAPREDUCE-4351-v5.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4295: - Status: Open (was: Patch Available) > RM crashes due to DNS issue > --- > > Key: MAPREDUCE-4295 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, > MAPREDUCE-4295.patch > > > we had a DNS outage and the RM crashed with the following backtrace: > 2012-05-29 19:17:34,492 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: java.net.UnknownHostException: > host.comat > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) > at > org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) >at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397896#comment-13397896 ] Hudson commented on MAPREDUCE-3889: --- Integrated in Hadoop-Common-trunk-Commit #2374 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2374/]) MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java > job client tries to use /tasklog interface, but that doesn't exist anymore > -- > > Key: MAPREDUCE-3889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 2.0.1-alpha, 3.0.0 >Reporter: Thomas Graves >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3889.patch, MAPREDUCE-3889.patch > > > if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when > running a job it tries to fetch task logs to print out on the client side > from a url like: > http://nodemanager:8080/tasklog?plaintext=true&attemptid=attempt_1329857083014_0003_r_00_0&filter=stdout > It always errors on this request with: Required param job, map and reduce > We saw this error when using distcp and the distcp failed. I'm not sure if it > is mandatory for distcp or just informational purposes. I'm guessing the > latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397895#comment-13397895 ] Thomas Graves commented on MAPREDUCE-4295: -- Thanks for the review Bobby. - I'll pull call to ContainerTokenIdentifier out - yes I meant to pass it. thanks for catching that. > RM crashes due to DNS issue > --- > > Key: MAPREDUCE-4295 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, > MAPREDUCE-4295.patch > > > we had a DNS outage and the RM crashed with the following backtrace: > 2012-05-29 19:17:34,492 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: java.net.UnknownHostException: > host.comat > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) > at > org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) >at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397892#comment-13397892 ] Hudson commented on MAPREDUCE-3889: --- Integrated in Hadoop-Hdfs-trunk-Commit #2444 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2444/]) MAPREDUCE-3889. job client tries to use /tasklog interface, but that doesn't exist anymore (Devaraj K via bobby) (Revision 1352330) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1352330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java > job client tries to use /tasklog interface, but that doesn't exist anymore > -- > > Key: MAPREDUCE-3889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 2.0.1-alpha, 3.0.0 >Reporter: Thomas Graves >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3889.patch, MAPREDUCE-3889.patch > > > if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when > running a job it tries to fetch task logs to print out on the client side > from a url like: > http://nodemanager:8080/tasklog?plaintext=true&attemptid=attempt_1329857083014_0003_r_00_0&filter=stdout > It always errors on this request with: Required param job, map and reduce > We saw this error when using distcp and the distcp failed. I'm not sure if it > is mandatory for distcp or just informational purposes. I'm guessing the > latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4266) remove Ant remnants from MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397884#comment-13397884 ] Thomas Graves commented on MAPREDUCE-4266: -- No, I mean instead of having a patch with a bunch of "-"'s have a shell script the committer runs to remove. I guess it doesn't matter to much. > remove Ant remnants from MR > --- > > Key: MAPREDUCE-4266 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4266 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 2.0.1-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.1-alpha > > Attachments: MAPREDUCE-4266.patch > > > Remove: > hadoop-mapreduce-project/src/* > hadoop-mapreduce-project/ivy/* > hadoop-mapreduce-project/build.xml > hadoop-mapreduce-project/ivy.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397882#comment-13397882 ] Robert Joseph Evans commented on MAPREDUCE-3889: The patch looks good to me +1. I am happy to check it in, but I would like to see a follow on JIRA to update the client to add back in the functionality. > job client tries to use /tasklog interface, but that doesn't exist anymore > -- > > Key: MAPREDUCE-3889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 2.0.1-alpha, 3.0.0 >Reporter: Thomas Graves >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3889.patch, MAPREDUCE-3889.patch > > > if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when > running a job it tries to fetch task logs to print out on the client side > from a url like: > http://nodemanager:8080/tasklog?plaintext=true&attemptid=attempt_1329857083014_0003_r_00_0&filter=stdout > It always errors on this request with: Required param job, map and reduce > We saw this error when using distcp and the distcp failed. I'm not sure if it > is mandatory for distcp or just informational purposes. I'm guessing the > latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397881#comment-13397881 ] Tom White commented on MAPREDUCE-3837: -- TestRecoveryManager and TestJobTrackerRestartWithLostTracker failed for me with this patch. Mayank - can you update them for this JIRA please? > Hadoop 22 Job tracker is not able to recover job in case of crash and after > that no user can submit job. > > > Key: MAPREDUCE-3837 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Fix For: 0.24.0, 0.22.1, 0.23.2 > > Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, > PATCH-HADOOP-1-MAPREDUCE-3837-2.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, > PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch > > > If job tracker is crashed while running , and there were some jobs are > running , so if job tracker's property mapreduce.jobtracker.restart.recover > is true then it should recover the job. > However the current behavior is as follows > jobtracker try to restore the jobs but it can not . And after that jobtracker > closes its handle to hdfs and nobody else can submit job. > Thanks, > Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4266) remove Ant remnants from MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397875#comment-13397875 ] Alejandro Abdelnur commented on MAPREDUCE-4266: --- you mean shelling out from Maven? like cmake does? > remove Ant remnants from MR > --- > > Key: MAPREDUCE-4266 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4266 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 2.0.1-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.1-alpha > > Attachments: MAPREDUCE-4266.patch > > > Remove: > hadoop-mapreduce-project/src/* > hadoop-mapreduce-project/ivy/* > hadoop-mapreduce-project/build.xml > hadoop-mapreduce-project/ivy.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4266) remove Ant remnants from MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397869#comment-13397869 ] Thomas Graves commented on MAPREDUCE-4266: -- sorry for the delay, pipes is now mavenized. Would this be better to do as a shell script? > remove Ant remnants from MR > --- > > Key: MAPREDUCE-4266 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4266 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 2.0.1-alpha >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.1-alpha > > Attachments: MAPREDUCE-4266.patch > > > Remove: > hadoop-mapreduce-project/src/* > hadoop-mapreduce-project/ivy/* > hadoop-mapreduce-project/build.xml > hadoop-mapreduce-project/ivy.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4358) Reducers are assigned containers before all maps are assigned containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397868#comment-13397868 ] Jason Lowe commented on MAPREDUCE-4358: --- The AM is supposed to be watching its headroom to see if it has enough resources for running maps and will even preempt (kill) reduce tasks to make room for map tasks when there isn't enough headroom. Which scheduler is being used for this test? If it's the FifoScheduler then I believe this is a duplicate of MAPREDUCE-4299. If the AM is being told an incorrectly computed headroom then it could think there is sufficient room to allocate containers for maps when in fact they've all been filled with reduce tasks. > Reducers are assigned containers before all maps are assigned containers > > > Key: MAPREDUCE-4358 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4358 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Ahmed Radwan > > Reducers start to get containers before all maps are. We have seen this issue > and it is problematic since if there is no avaialable resources for the > remaining maps, the job will just stall where reducers are waiting for > mappers which are unable to start because there is no containers available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397865#comment-13397865 ] Robert Joseph Evans commented on MAPREDUCE-4295: The change looks good to me. I just have two minor comments. # It is BuilderUtils.newContainerToken that could throw the IllegalArgumentException because of a DNS outage. It might be good to move the creation of the ContainerTokenIdentifier out of the try block so we don't accidentally treat something else as a DNS issue. This is mostly because IllegalArgumentException really does not indicate that there was a problem resolving a host name, so I would prefer to reduce the chance of handling the error incorrectly. # Do we want to log the exception as a string or do we want to pass in the exception as an argument to the log so we get the full stack trace? In the case of a DNS outage we may not want to fill up the logs with useless messages, but if there is a different error happening I would prefer to have more information to be able to debug it. > RM crashes due to DNS issue > --- > > Key: MAPREDUCE-4295 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, > MAPREDUCE-4295.patch > > > we had a DNS outage and the RM crashed with the following backtrace: > 2012-05-29 19:17:34,492 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: java.net.UnknownHostException: > host.comat > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) > at > org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) >at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3868) Reenable Raid
[ https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397859#comment-13397859 ] Thomas Graves commented on MAPREDUCE-3868: -- this also introduced a bunch of javadoc warnings - see HDFS-3550 if you don't have findbugs to exclude remove that from the pom file. If you do put the file in svn. You can test by running test-patch manually. > Reenable Raid > - > > Key: MAPREDUCE-3868 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Reporter: Scott Chen >Assignee: Weiyan Wang > Attachments: MAPREDUCE-3868-1.patch, MAPREDUCE-3868-2.patch, > MAPREDUCE-3868-3.patch, MAPREDUCE-3868.patch, MAPREDUCE-3868v1.patch, > MAPREDUCE-3868v1.sh > > > Currently Raid is outdated and not compiled. Make it compile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4320) gridmix mainClass wrong in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397856#comment-13397856 ] Hadoop QA commented on MAPREDUCE-4320: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532757/MAPREDUCE-4320.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-gridmix. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2487//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2487//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-gridmix.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2487//console This message is automatically generated. > gridmix mainClass wrong in pom.xml > -- > > Key: MAPREDUCE-4320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4320 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4320.patch, MAPREDUCE-4320.patch, > MAPREDUCE-4320.patch > > > when trying to run gridmix its actually trying to run > org.apache.hadoop.tools.HadoopArchives. > the pom.xml needs to be fixed to have correct mainClass: > org.apache.hadoop.mapred.gridmix.Gridmix -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4320) gridmix mainClass wrong in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397845#comment-13397845 ] Robert Joseph Evans commented on MAPREDUCE-4320: The change looks good to me +1. I see no way that TestResourceUsageEmulators could fail because of this change. > gridmix mainClass wrong in pom.xml > -- > > Key: MAPREDUCE-4320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4320 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4320.patch, MAPREDUCE-4320.patch, > MAPREDUCE-4320.patch > > > when trying to run gridmix its actually trying to run > org.apache.hadoop.tools.HadoopArchives. > the pom.xml needs to be fixed to have correct mainClass: > org.apache.hadoop.mapred.gridmix.Gridmix -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397843#comment-13397843 ] Hadoop QA commented on MAPREDUCE-4295: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532752/MAPREDUCE-4295.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2486//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2486//console This message is automatically generated. > RM crashes due to DNS issue > --- > > Key: MAPREDUCE-4295 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, > MAPREDUCE-4295.patch > > > we had a DNS outage and the RM crashed with the following backtrace: > 2012-05-29 19:17:34,492 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: java.net.UnknownHostException: > host.comat > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) > at > org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) >at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4320) gridmix mainClass wrong in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4320: - Status: Patch Available (was: Open) > gridmix mainClass wrong in pom.xml > -- > > Key: MAPREDUCE-4320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4320 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4320.patch, MAPREDUCE-4320.patch, > MAPREDUCE-4320.patch > > > when trying to run gridmix its actually trying to run > org.apache.hadoop.tools.HadoopArchives. > the pom.xml needs to be fixed to have correct mainClass: > org.apache.hadoop.mapred.gridmix.Gridmix -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4320) gridmix mainClass wrong in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4320: - Attachment: MAPREDUCE-4320.patch I wasn't able to reproduce the test failure. Trying again. > gridmix mainClass wrong in pom.xml > -- > > Key: MAPREDUCE-4320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4320 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4320.patch, MAPREDUCE-4320.patch, > MAPREDUCE-4320.patch > > > when trying to run gridmix its actually trying to run > org.apache.hadoop.tools.HadoopArchives. > the pom.xml needs to be fixed to have correct mainClass: > org.apache.hadoop.mapred.gridmix.Gridmix -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4300) OOM in AM can turn it into a zombie.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397833#comment-13397833
]
Robert Joseph Evans commented on MAPREDUCE-4300:
I would like some feedback on this. I see several different threads that got
OOM errors in them. None of them handled the errors gracefully and shut down.
Arguably I am not sure that the Metrics System, HDFS library, or the default
speculator thread should try to shut down the process in such a case. I
personally would prefer to see us install a defaultUncaughtExceptionHandler for
all threads. If a thread throws a Throwable that is not caught implying that
we did not expect to see it we can try and shut down the process cleanly.
I don't really want to start putting catch(Throwable t) everywhere it is too
easy to miss a single thread and then be back in the same boat.
> OOM in AM can turn it into a zombie.
>
>
> Key: MAPREDUCE-4300
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4300
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 0.23.3
>Reporter: Robert Joseph Evans
>
> It looks like 4 threads in the AM died with OOM but not the one pinging the
> RM.
> stderr for this AM
> {noformat}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
> org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> May 30, 2012 4:49:55 AM
> com.google.inject.servlet.InternalServletModule$BackwardsCompatibleServletContextProvider
> get
> WARNING: You are attempting to use a deprecated API (specifically, attempting
> to @Inject ServletContext inside an eagerly created singleton. While we allow
> this for backwards compatibility, be warned that this MAY have unexpected
> behavior if you have more than one injector (with ServletModule) running in
> the same JVM. Please consult the Guice documentation at
> http://code.google.com/p/google-guice/wiki/Servlets for more information.
> May 30, 2012 4:49:55 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider
> class
> May 30, 2012 4:49:55 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a
> provider class
> May 30, 2012 4:49:55 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as
> a root resource class
> May 30, 2012 4:49:55 AM
> com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
> INFO: Initiating Jersey application, version 'Jersey: 1.8 06/24/2011 12:17 PM'
> May 30, 2012 4:49:55 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver
> to GuiceManagedComponentProvider with the scope "Singleton"
> May 30, 2012 4:49:56 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to
> GuiceManagedComponentProvider with the scope "Singleton"
> May 30, 2012 4:49:56 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to
> GuiceManagedComponentProvider with the scope "PerRequest"
> Exception in thread "ResponseProcessor for block
> BP-1114822160--1322528669066:blk_-6528896407411719649_34227308"
> java.lang.OutOfMemoryError: Java heap space
> at com.google.protobuf.CodedInputStream.(CodedInputStream.java:538)
> at
> com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:201)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:738)
> at
> org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos$PipelineAckProto.parseFrom(DataTransferProtos.java:7287)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:95)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:656)
> Exception in thread "DefaultSpeculator background processing"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.getTasks(
[jira] [Updated] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4295: - Attachment: MAPREDUCE-4295.patch fix findbugs issue. > RM crashes due to DNS issue > --- > > Key: MAPREDUCE-4295 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, > MAPREDUCE-4295.patch > > > we had a DNS outage and the RM crashed with the following backtrace: > 2012-05-29 19:17:34,492 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: java.net.UnknownHostException: > host.comat > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) > at > org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) >at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4295: - Status: Patch Available (was: Open) > RM crashes due to DNS issue > --- > > Key: MAPREDUCE-4295 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch, > MAPREDUCE-4295.patch > > > we had a DNS outage and the RM crashed with the following backtrace: > 2012-05-29 19:17:34,492 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: java.net.UnknownHostException: > host.comat > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) > at > org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) >at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4358) Reducers are assigned containers before all maps are assigned containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397825#comment-13397825 ] Robert Joseph Evans commented on MAPREDUCE-4358: Isn't that what mapreduce.job.reduce.slowstart.completedmaps is for? If you don't want any reducers to run until all of the maps have finished then you want to set it to 1.0, not the 0.05 that is the default. I think part of the issue is that the default value for mapreduce.job.reduce.slowstart.completedmaps is still set for when map and reduce slots were completely separate. Perhaps this config does not make since any more now that reduce tasks can block map tasks from running. Or perhaps we need another config so that the AM will not fill more than X% of the queue with reduces until all map tasks have completed. We have taken the rout of setting the slowstart to 1.0 even on our 1.0.2 clusters because it improves the cluster utilization and we have not seen much of a hit to the end to end time of our jobs. > Reducers are assigned containers before all maps are assigned containers > > > Key: MAPREDUCE-4358 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4358 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Ahmed Radwan > > Reducers start to get containers before all maps are. We have seen this issue > and it is problematic since if there is no avaialable resources for the > remaining maps, the job will just stall where reducers are waiting for > mappers which are unable to start because there is no containers available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4358) Reducers are assigned containers before all maps are assigned containers
Ahmed Radwan created MAPREDUCE-4358: --- Summary: Reducers are assigned containers before all maps are assigned containers Key: MAPREDUCE-4358 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4358 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Ahmed Radwan Reducers start to get containers before all maps are. We have seen this issue and it is problematic since if there is no avaialable resources for the remaining maps, the job will just stall where reducers are waiting for mappers which are unable to start because there is no containers available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397804#comment-13397804 ] Andrew Ferguson commented on MAPREDUCE-4351: .bq Oh also please look into the findbugs and javadoc warnings. The findbugs warning is spurious; it's from existing code and triggered by the rename of the file. (the cast happens inside a switch block on monitoringEvent.getType()) Not sure about the javadoc warnings yet. > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397801#comment-13397801 ] Andrew Ferguson commented on MAPREDUCE-4351: Hi Robert, Thanks for looking at the patch. bq. I am a bit confused as to why the creation of the ContainersMonitor was moved from ContainerManagerImpl to NodeManager. The NodeManager does not appear to have any need for it. When I put loading ContainersMonitor into ContainerManagerImpl, a number of tests failed. It seemed to want to be added as a service before ContainerManagerImpl was created, but I don't have a full understanding of the services and how they relate, so this may be have been the wrong move. This way, it also matches the way ContainerExecutor is loaded, or how other services such as the NodeStatusUpdater are started. bq. I would also like to see some more documentation about how a ContainersMonitor is supposed to behave. The only API in there is setup. It would be nice to be able to document what events a ContaiersMonitor is expected to handle (START_MONITORING_CONTAINER, STOP_MONITORING_CONTAINER) and how it may optionally stop a misbehaving container ContainerKillEvent. Great point, thanks. I will add documentation. Andrew > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397795#comment-13397795 ] Robert Joseph Evans commented on MAPREDUCE-4351: @Arun, Having cgroups will block the container in, but there are other use cases. # There will be some OSs we want to support that do not have very good process group isolation so having this pluggable allows us to put in a NullMonitor for cgroups and a default/polling monitor for older versions of Linux. # There are also cases where we may only want to monitor the resource and not shoot the process. There was some discussion about providing feedback to the scheduler about what resources each container is actually using so that it can potentially over commit. Even in the case of cgroups we would need something to be able to get that information and return it to the RM. There is some work that needs to go into figuring out exactly how we want to monitor resources vs. enforce resources, and how we want to support this for different resources. I did not see much of an issue with putting in a small amount of refactoring so that others can start experimenting with the stipulation that everything may change in the future, @Unstable. If you disagree I am fine with holding off on it and starting that discussion now. Either here, on MAPREDUCE-4256, or perhaps in a new rollup JIRA for the full feature. > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397794#comment-13397794 ] Andrew Ferguson commented on MAPREDUCE-4351: Hi Arun, I initially considered that approach (having ContainersMonitor go away), but decided against it for the following reasons: - Because the initial configuration of cgroups requires root access, some users may not be able to use them. Therefore, it makes sense to keep the current ContainersMonitor, which monitors memory usage and does its own OOM killing. - Maintaining the separation between the ContainerExecutor (which just does launching) and the ContainersMonitor (which just does enforcement) lowers the amount of code overlap for supporting each permutation: secure executor with/without cgroups, default executor with/without cgroups, a ContainersMonitor with taskset for CPU instead of cgroups, etc. - With this approach, the ContainersMonitor continues to receive the ContainerStopMonitoringEvent event, which allows it to delete cgroups which are no longer in use. - Finally, the current ContainersMonitor does not start enforcing the memory limit until after a short delay because of the potential to double-count resources when the JVM does a fork()+exec() -- see comment above isProcessTreeOverLimit(). Keeping resource enforcement separate from process launching gives us the flexibility to delay the start of enforcement if needed. do these reasons make sense? I'm currently writing a "CgroupsContainersMonitor" to complement the "DefaultContainersMonitor," and which plugs-in using this patch. As you point out, this CgroupsContainerMonitor doesn't do much work: it just configures cgroups and places processes in them, allowing the kernel to do the actual enforcement work. thanks! Andrew > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397790#comment-13397790 ] Hadoop QA commented on MAPREDUCE-4355: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532743/MR-4355_mr2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2485//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2485//console This message is automatically generated. > Add JobStatus getJobStatus(JobID) to JobClient. > --- > > Key: MAPREDUCE-4355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv1, mrv2 >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: MR-4355_mr2.patch > > > To read the start-time of a particular job, one should not need to > getAllJobs() and iterate through them. > getJob(JobID) returns RunningJob, which doesn't hold the job's start time. > Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397773#comment-13397773 ] Arun C Murthy commented on MAPREDUCE-4351: -- I'm sorry to come in late... I'm a little confused by this approach, so please bear with me. It seems to me that ContainersMonitory should actually go away once we goto cgroups etc., since the cgroup itself would 'box' the process in, there-by removing the need for any monitoring from NodeManager. If so, this would be not required at all? Thus, once we go via the cgroups route, the LCE should do the necessary work to put the process in the cgroup. This means that we will need different *versions* of LCE for rhel5/centos5, rhel6/centos6 etc. > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4355: Release Note: - Augments the API - JobClient.getJobStatus(JobID) - Renames existing files - TestJobClientGetJob to TestJobClient Hadoop Flags: Incompatible change Status: Patch Available (was: Open) Submitting to make sure Hudson doesn't complain > Add JobStatus getJobStatus(JobID) to JobClient. > --- > > Key: MAPREDUCE-4355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv1, mrv2 >Affects Versions: 2.0.0-alpha, 1.0.3 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: MR-4355_mr2.patch > > > To read the start-time of a particular job, one should not need to > getAllJobs() and iterate through them. > getJob(JobID) returns RunningJob, which doesn't hold the job's start time. > Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of ProcfsBasedProcessTree for Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397766#comment-13397766 ] Ivan Mitic commented on MAPREDUCE-4203: --- +1, change looks good, thanks Bikas. > Create equivalent of ProcfsBasedProcessTree for Windows > --- > > Key: MAPREDUCE-4203 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: MAPREDUCE-4203.branch-1-win.1.patch, > MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, > MAPREDUCE-4203.branch-1-win.4.patch, MAPREDUCE-4203.patch, test.cpp > > > ProcfsBasedProcessTree is used by the TaskTracker to get process information > like memory and cpu usage. This information is used to manage resources etc. > The current implementation is based on Linux procfs functionality and hence > does not work on other platforms, specifically windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4357) Snappy Codec does not load properly when m/r job is run in "uber" mode
Jeff Lord created MAPREDUCE-4357:
Summary: Snappy Codec does not load properly when m/r job is run
in "uber" mode
Key: MAPREDUCE-4357
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4357
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Jeff Lord
# sudo -u hdfs hadoop jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.0.0-cdh4.0.0-tests.jar
TestDFSIO -write
12/06/01 18:17:11 INFO fs.TestDFSIO: TestDFSIO.0.0.6
12/06/01 18:17:11 INFO fs.TestDFSIO: nrFiles = 1
12/06/01 18:17:11 INFO fs.TestDFSIO: fileSize (MB) = 1.0
12/06/01 18:17:11 INFO fs.TestDFSIO: bufferSize = 100
12/06/01 18:17:11 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
12/06/01 18:17:11 INFO fs.TestDFSIO: creating control file: 1048576 bytes, 1
files
12/06/01 18:17:12 INFO fs.TestDFSIO: created control files for: 1 files
12/06/01 18:17:12 INFO mapred.FileInputFormat: Total input paths to process : 1
12/06/01 18:17:12 INFO mapreduce.JobSubmitter: number of splits:1
12/06/01 18:17:12 WARN conf.Configuration: mapred.jar is deprecated. Instead,
use mapreduce.job.jar
12/06/01 18:17:12 WARN conf.Configuration: mapred.reduce.tasks is deprecated.
Instead, use mapreduce.job.reduces
12/06/01 18:17:12 WARN conf.Configuration: mapred.output.value.class is
deprecated. Instead, use mapreduce.job.output.value.class
12/06/01 18:17:12 WARN conf.Configuration: mapred.job.name is deprecated.
Instead, use mapreduce.job.name
12/06/01 18:17:12 WARN conf.Configuration: mapred.input.dir is deprecated.
Instead, use mapreduce.input.fileinputformat.inputdir
12/06/01 18:17:12 WARN conf.Configuration: mapred.output.dir is deprecated.
Instead, use mapreduce.output.fileoutputformat.outputdir
12/06/01 18:17:12 WARN conf.Configuration: mapred.map.tasks is deprecated.
Instead, use mapreduce.job.maps
12/06/01 18:17:12 WARN conf.Configuration: mapred.output.key.class is
deprecated. Instead, use mapreduce.job.output.key.class
12/06/01 18:17:12 WARN conf.Configuration: mapred.working.dir is deprecated.
Instead, use mapreduce.job.working.dir
12/06/01 18:17:12 INFO mapred.ResourceMgrDelegate: Submitted application
application_1338599410922_0004 to ResourceManager at /0.0.0.0:8032
12/06/01 18:17:12 INFO mapreduce.Job: The url to track the job:
http://localhost:8088/proxy/application_1338599410922_0004/
12/06/01 18:17:12 INFO mapreduce.Job: Running job: job_1338599410922_0004
12/06/01 18:17:17 INFO mapreduce.Job: Job job_1338599410922_0004 running in
uber mode : true
12/06/01 18:17:17 INFO mapreduce.Job: map 0% reduce 0%
12/06/01 18:17:17 INFO mapreduce.Job: Job job_1338599410922_0004 failed with
state FAILED due to:
12/06/01 18:17:17 INFO mapreduce.Job: Counters: 11
Job Counters
Failed map tasks=1
Failed reduce tasks=1
Launched map tasks=1
Launched reduce tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=2456
Total time spent by all reduces in occupied slots (ms)=136
TOTAL_LAUNCHED_UBERTASKS=2
NUM_UBER_SUBMAPS=1
NUM_UBER_SUBREDUCES=1
NUM_FAILED_UBERTASKS=2
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:883)
at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:340)
at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:321)
at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:520)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:445)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:112)
at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Attaching my mapred-site.xml as well.
I have snappy compression enabled but I never see it get loaded in the logs.
cat /etc/hadoop/conf.pseudo/mapred-site.xml
mapred.job.tracker
localhost:8021
mapreduce.framework.name
yarn
To set the value of tmp directory for map and reduce
tasks.
mapreduce.task.tmp.dir
/var/lib/hadoop-mapreduce/cache/${user.name}/tasks
mapreduce.job.ubertask.enable
true
Run very small j
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of ProcfsBasedProcessTree for Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397763#comment-13397763 ] Chuan Liu commented on MAPREDUCE-4203: -- +1 Thanks! > Create equivalent of ProcfsBasedProcessTree for Windows > --- > > Key: MAPREDUCE-4203 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: MAPREDUCE-4203.branch-1-win.1.patch, > MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, > MAPREDUCE-4203.branch-1-win.4.patch, MAPREDUCE-4203.patch, test.cpp > > > ProcfsBasedProcessTree is used by the TaskTracker to get process information > like memory and cpu usage. This information is used to manage resources etc. > The current implementation is based on Linux procfs functionality and hence > does not work on other platforms, specifically windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4355) Add JobStatus getJobStatus(JobID) to JobClient.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-4355: Attachment: MR-4355_mr2.patch Uploading a patch for MR2: MR-4355_mr2.patch. The patch - implements JobClient.getJobStatus(JobID) - renames TestJobClientGetJob to TestJobClient - adds a test for getJobStatus to the renamed test file > Add JobStatus getJobStatus(JobID) to JobClient. > --- > > Key: MAPREDUCE-4355 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv1, mrv2 >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: MR-4355_mr2.patch > > > To read the start-time of a particular job, one should not need to > getAllJobs() and iterate through them. > getJob(JobID) returns RunningJob, which doesn't hold the job's start time. > Hence, we need to add getJobStatus(JobID) to the API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4203) Create equivalent of ProcfsBasedProcessTree for Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-4203: -- Attachment: MAPREDUCE-4203.branch-1-win.4.patch Attaching new patch with error mesg change and wide char usage. > Create equivalent of ProcfsBasedProcessTree for Windows > --- > > Key: MAPREDUCE-4203 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: MAPREDUCE-4203.branch-1-win.1.patch, > MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, > MAPREDUCE-4203.branch-1-win.4.patch, MAPREDUCE-4203.patch, test.cpp > > > ProcfsBasedProcessTree is used by the TaskTracker to get process information > like memory and cpu usage. This information is used to manage resources etc. > The current implementation is based on Linux procfs functionality and hence > does not work on other platforms, specifically windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397743#comment-13397743 ] Robert Joseph Evans commented on MAPREDUCE-4351: Oh also please look into the findbugs and javadoc warnings. > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[
https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397742#comment-13397742
]
Robert Joseph Evans commented on MAPREDUCE-4351:
I have been looking a little more closely at the patch. I am a bit confused as
to why the creation of the ContainersMonitor was moved from
ContainerManagerImpl to NodeManager. The NodeManager does not appear to have
any need for it.
to Answer your question {quote}One point I would like feedback on:
ContainerManagerImpl's getContainersMonitor() is only used in NodeManager's
init() as part of the call to createWebServer() — but with this patch, the
ContainersMonitor is now created locally and passed to the ContainerManager in
NodeManager's init(). Does it make sense to remove this function?{quote} I
don't really see a need to change it at this point, especially if we push the
creation of the ContainersMonitor back into the ContainerManagerImpl.
I would also like to see some more documentation about how a ContainersMonitor
is supposed to behave. The only API in there is setup. It would be nice to be
able to document what events a ContaiersMonitor is expected to handle
(START_MONITORING_CONTAINER, STOP_MONITORING_CONTAINER) and how it may
optionally stop a misbehaving container ContainerKillEvent.
> Make ContainersMonitor pluggable
>
>
> Key: MAPREDUCE-4351
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, nodemanager
>Reporter: Andrew Ferguson
>Assignee: Andrew Ferguson
> Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch,
> MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch
>
>
> Make the existing ContainersManager pluggable, just as the ContainerExecutor
> is currently. This will allow us to add container resource enforcement using
> other techniques (such as cgroups) in an extensible fashion.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397684#comment-13397684 ] Hadoop QA commented on MAPREDUCE-4351: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532727/MAPREDUCE-4351-v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2484//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2484//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2484//console This message is automatically generated. > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4295: - Status: Open (was: Patch Available) fixing findbug error > RM crashes due to DNS issue > --- > > Key: MAPREDUCE-4295 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch > > > we had a DNS outage and the RM crashed with the following backtrace: > 2012-05-29 19:17:34,492 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: java.net.UnknownHostException: > host.comat > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) > at > org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) >at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4320) gridmix mainClass wrong in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4320: - Status: Open (was: Patch Available) investigating test failure > gridmix mainClass wrong in pom.xml > -- > > Key: MAPREDUCE-4320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4320 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4320.patch, MAPREDUCE-4320.patch > > > when trying to run gridmix its actually trying to run > org.apache.hadoop.tools.HadoopArchives. > the pom.xml needs to be fixed to have correct mainClass: > org.apache.hadoop.mapred.gridmix.Gridmix -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4203) Create equivalent of ProcfsBasedProcessTree for Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397670#comment-13397670 ] Bikas Saha commented on MAPREDUCE-4203: --- ok. for this specific case I will just print an error message inline. > Create equivalent of ProcfsBasedProcessTree for Windows > --- > > Key: MAPREDUCE-4203 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4203 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: MAPREDUCE-4203.branch-1-win.1.patch, > MAPREDUCE-4203.branch-1-win.2.patch, MAPREDUCE-4203.branch-1-win.3.patch, > MAPREDUCE-4203.patch, test.cpp > > > ProcfsBasedProcessTree is used by the TaskTracker to get process information > like memory and cpu usage. This information is used to manage resources etc. > The current implementation is based on Linux procfs functionality and hence > does not work on other platforms, specifically windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4351: - Attachment: MAPREDUCE-4351-v4.patch reattaching same patch to kick jenkins as its fixed now. > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4267) mavenize pipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397649#comment-13397649 ] Thomas Graves commented on MAPREDUCE-4267: -- I installed the 32 bit version of the library, should be fixed now. > mavenize pipes > -- > > Key: MAPREDUCE-4267 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4267 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 > > Attachments: MAPREDUCE-4267.001.rm.patch, > MAPREDUCE-4267.001.trimmed.patch, MAPREDUCE-4267.002.trimmed.patch, > MAPREDUCE-4267.patch, MAPREDUCE-4267.patch, MAPREDUCE-4267.patch, > MAPREDUCE-4267.sh > > > We are still building pipes out of the old mrv1 directories using ant. Move > it over to the mrv2 dir structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397646#comment-13397646 ] Hadoop QA commented on MAPREDUCE-4295: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532722/MAPREDUCE-4295.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2483//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2483//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2483//console This message is automatically generated. > RM crashes due to DNS issue > --- > > Key: MAPREDUCE-4295 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch > > > we had a DNS outage and the RM crashed with the following backtrace: > 2012-05-29 19:17:34,492 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: java.net.UnknownHostException: > host.comat > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) > at > org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) >at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4295) RM crashes due to DNS issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4295: - Attachment: MAPREDUCE-4295.patch > RM crashes due to DNS issue > --- > > Key: MAPREDUCE-4295 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4295 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4295.patch, MAPREDUCE-4295.patch > > > we had a DNS outage and the RM crashed with the following backtrace: > 2012-05-29 19:17:34,492 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type NODE_UPDATE to the scheduler > java.lang.IllegalArgumentException: java.net.UnknownHostException: > host.comat > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:430) > at > org.apache.hadoop.yarn.util.BuilderUtils.newContainerToken(BuilderUtils.java:261) >at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1184) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1167) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1213) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4320) gridmix mainClass wrong in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397623#comment-13397623 ] Hadoop QA commented on MAPREDUCE-4320: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12532717/MAPREDUCE-4320.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-tools/hadoop-gridmix: org.apache.hadoop.mapred.gridmix.TestResourceUsageEmulators +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2482//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2482//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-gridmix.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2482//console This message is automatically generated. > gridmix mainClass wrong in pom.xml > -- > > Key: MAPREDUCE-4320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4320 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4320.patch, MAPREDUCE-4320.patch > > > when trying to run gridmix its actually trying to run > org.apache.hadoop.tools.HadoopArchives. > the pom.xml needs to be fixed to have correct mainClass: > org.apache.hadoop.mapred.gridmix.Gridmix -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4320) gridmix mainClass wrong in pom.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4320: - Attachment: MAPREDUCE-4320.patch rekicking jenkins. > gridmix mainClass wrong in pom.xml > -- > > Key: MAPREDUCE-4320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4320 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4320.patch, MAPREDUCE-4320.patch > > > when trying to run gridmix its actually trying to run > org.apache.hadoop.tools.HadoopArchives. > the pom.xml needs to be fixed to have correct mainClass: > org.apache.hadoop.mapred.gridmix.Gridmix -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4267) mavenize pipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397548#comment-13397548 ] Robert Joseph Evans commented on MAPREDUCE-4267: It looks like this broke the build. It seems that the build servers do not have a 32bit version of libstdc++. > mavenize pipes > -- > > Key: MAPREDUCE-4267 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4267 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 > > Attachments: MAPREDUCE-4267.001.rm.patch, > MAPREDUCE-4267.001.trimmed.patch, MAPREDUCE-4267.002.trimmed.patch, > MAPREDUCE-4267.patch, MAPREDUCE-4267.patch, MAPREDUCE-4267.patch, > MAPREDUCE-4267.sh > > > We are still building pipes out of the old mrv1 directories using ant. Move > it over to the mrv2 dir structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4354) Performance improvement with compressor object reinit restriction
[ https://issues.apache.org/jira/browse/MAPREDUCE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397546#comment-13397546 ] Robert Joseph Evans commented on MAPREDUCE-4354: Wouldn't it be cleaner to stub out reinit for LZO if reinit is not needed for it? > Performance improvement with compressor object reinit restriction > - > > Key: MAPREDUCE-4354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4354 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: performance >Affects Versions: 0.20.205.0 >Reporter: Ankit Kamboj >Priority: Minor > Labels: performance > Fix For: 0.20.205.0 > > Attachments: codec_reinit_diff > > > HADOOP-5879 patch aimed at picking the conf (instead of default) settings for > GzipCodec. It also involved re-initializing the recycled compressor object. > On our performance tests, this re-initialization led to performance > degradation of 15% for LzoCodec because re-initialization for Lzo involves > reallocation of buffers. LzoCodec takes the initial settings from config so > it is not necessary to re-initialize it. This patch checks for the codec > class and calls reinit only if the codec class is Gzip. This led to > significant performance improvement of 15% for LzoCodec. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4306) Problem running Distributed Shell applications as a user other than the one started the daemons
[ https://issues.apache.org/jira/browse/MAPREDUCE-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397529#comment-13397529 ] Hudson commented on MAPREDUCE-4306: --- Integrated in Hadoop-Mapreduce-trunk #1115 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1115/]) MAPREDUCE-4306. Fix distributed shell to work with users other than the one running the daemons. (Contributed by Ahmed Radwan) (Revision 1351876) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351876 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java > Problem running Distributed Shell applications as a user other than the one > started the daemons > --- > > Key: MAPREDUCE-4306 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4306 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Fix For: 2.0.1-alpha > > Attachments: MAPREDUCE-4306.patch, MAPREDUCE-4306_rev2.patch, > MAPREDUCE-4306_rev3.patch > > > Using the tarball, if you start the yarn daemons using one user and then > switch to a different user. You can successfully run MR jobs, but DS jobs > fail to run. Only able to run DS jobs using the user who started the daemons. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4267) mavenize pipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397530#comment-13397530 ] Hudson commented on MAPREDUCE-4267: --- Integrated in Hadoop-Mapreduce-trunk #1115 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1115/]) MAPREDUCE-4267. mavenize pipes (tgraves via bobby) (Revision 1351818) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351818 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/c++/pipes * /hadoop/common/trunk/hadoop-mapreduce-project/src/c++/utils * /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/pipes * /hadoop/common/trunk/hadoop-tools/hadoop-pipes * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/.autom4te.cfg * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/Makefile.am * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/configure.ac * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/.autom4te.cfg * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/Makefile.am * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/configure.ac * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/.autom4te.cfg * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/Makefile.am * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/configure.ac * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/m4/hadoop_utils.m4 * /hadoop/common/trunk/hadoop-tools/hadoop-tools-dist/pom.xml * /hadoop/common/trunk/hadoop-tools/pom.xml > mavenize pipes > -- > > Key: MAPREDUCE-4267 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4267 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 > > Attachments: MAPREDUCE-4267.001.rm.patch, > MAPREDUCE-4267.001.trimmed.patch, MAPREDUCE-4267.002.trimmed.patch, > MAPREDUCE-4267.patch, MAPREDUCE-4267.patch, MAPREDUCE-4267.patch, > MAPREDUCE-4267.sh > > > We are still building pipes out of the old mrv1 directories using ant. Move > it over to the mrv2 dir structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4270) data_join test classes are in the wrong packge
[ https://issues.apache.org/jira/browse/MAPREDUCE-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397532#comment-13397532 ] Hudson commented on MAPREDUCE-4270: --- Integrated in Hadoop-Mapreduce-trunk #1115 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1115/]) MAPREDUCE-4270. Move the data_join test classes to the correct path. (Contributed by Thomas Graves) (Revision 1351869) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351869 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleTaggedMapOutput.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/SampleTaggedMapOutput.java > data_join test classes are in the wrong packge > -- > > Key: MAPREDUCE-4270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4270 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, trunk >Reporter: Brock Noland >Assignee: Thomas Graves > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4270.sh > > > There are three Sample*.java files in this directory > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/ > but they should be in > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/ > based on their package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4341) add types to capacity scheduler properties documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397531#comment-13397531 ] Hudson commented on MAPREDUCE-4341: --- Integrated in Hadoop-Mapreduce-trunk #1115 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1115/]) MAPREDUCE-4341. add types to capacity scheduler properties documentation (Karthik Kambatla via tgraves). (Revision 1351762) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351762 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm > add types to capacity scheduler properties documentation > > > Key: MAPREDUCE-4341 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4341 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched, mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Karthik Kambatla > Fix For: 0.23.3 > > Attachments: MR-4341.patch > > > MAPREDUCE-4311 is changing capacity/max capacity configuration to be floats. > We should document that in the capacity scheduler properties docs > (http://hadoop.apache.org/common/docs/r0.23.1/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Configuration). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4311) Capacity scheduler.xml does not accept decimal values for capacity and maximum-capacity settings
[ https://issues.apache.org/jira/browse/MAPREDUCE-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397533#comment-13397533 ] Hudson commented on MAPREDUCE-4311: --- Integrated in Hadoop-Mapreduce-trunk #1115 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1115/]) MAPREDUCE-4311. Capacity scheduler.xml does not accept decimal values for capacity and maximum-capacity settings (Karthik Kambatla via tgraves) (Revision 1351700) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351700 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > Capacity scheduler.xml does not accept decimal values for capacity and > maximum-capacity settings > > > Key: MAPREDUCE-4311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4311 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched, mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Karthik Kambatla > Fix For: 0.23.3 > > Attachments: MR-4311.patch > > > if capacity scheduler capacity or max capacity set with decimal it errors: > - Error starting ResourceManager > java.lang.NumberFormatException: For input string: "10.5" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) > at java.lang.Integer.parseInt(Integer.java:458) > at java.lang.Integer.parseInt(Integer.java:499) > at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:713) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getCapacity(CapacitySchedulerConfiguration.java:147) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:147) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:297) > at > 0.20 used to take decimal and this could be an issue on large clusters that > would have queues with small allocations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397521#comment-13397521 ] Thomas Graves commented on MAPREDUCE-4351: -- this is from the mavenization of pipes. Looks like build machine is missing 32 bit library. I am working on fixing. > Make ContainersMonitor pluggable > > > Key: MAPREDUCE-4351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, nodemanager >Reporter: Andrew Ferguson >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, > MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch > > > Make the existing ContainersManager pluggable, just as the ContainerExecutor > is currently. This will allow us to add container resource enforcement using > other techniques (such as cgroups) in an extensible fashion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4311) Capacity scheduler.xml does not accept decimal values for capacity and maximum-capacity settings
[ https://issues.apache.org/jira/browse/MAPREDUCE-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397468#comment-13397468 ] Hudson commented on MAPREDUCE-4311: --- Integrated in Hadoop-Hdfs-0.23-Build #292 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/292/]) merge -r 1351702:1351703 from branch-2. FIXES: MAPREDUCE-4311 (Revision 1351704) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351704 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > Capacity scheduler.xml does not accept decimal values for capacity and > maximum-capacity settings > > > Key: MAPREDUCE-4311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4311 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched, mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Karthik Kambatla > Fix For: 0.23.3 > > Attachments: MR-4311.patch > > > if capacity scheduler capacity or max capacity set with decimal it errors: > - Error starting ResourceManager > java.lang.NumberFormatException: For input string: "10.5" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) > at java.lang.Integer.parseInt(Integer.java:458) > at java.lang.Integer.parseInt(Integer.java:499) > at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:713) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getCapacity(CapacitySchedulerConfiguration.java:147) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:147) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:297) > at > 0.20 used to take decimal and this could be an issue on large clusters that > would have queues with small allocations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4267) mavenize pipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397465#comment-13397465 ] Hudson commented on MAPREDUCE-4267: --- Integrated in Hadoop-Hdfs-0.23-Build #292 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/292/]) svn merge -c 1351818. FIXESL MAPREDUCE-4267. mavenize pipes (tgraves via bobby) (Revision 1351833) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351833 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++/pipes * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++/utils * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples/pipes * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-pipes * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-pipes/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-pipes/src * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-pipes/src/CMakeLists.txt * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-pipes/src/main * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-pipes/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-pipes/src/main/native/examples * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-pipes/src/main/native/examples/README.txt * /hadoop/common/branches/branch
[jira] [Commented] (MAPREDUCE-4270) data_join test classes are in the wrong packge
[ https://issues.apache.org/jira/browse/MAPREDUCE-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397467#comment-13397467 ] Hudson commented on MAPREDUCE-4270: --- Integrated in Hadoop-Hdfs-0.23-Build #292 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/292/]) Merge MAPREDUCE-4270 from trunk. Move the data_join test classes to the correct path. (Contributed by Thomas Graves) (Revision 1351871) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351871 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-datajoin/src/test/java/README.txt * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinMapper.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinReducer.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-datajoin/src/test/java/SampleTaggedMapOutput.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/README.txt * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/SampleDataJoinMapper.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/SampleDataJoinReducer.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/SampleTaggedMapOutput.java > data_join test classes are in the wrong packge > -- > > Key: MAPREDUCE-4270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4270 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, trunk >Reporter: Brock Noland >Assignee: Thomas Graves > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4270.sh > > > There are three Sample*.java files in this directory > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/ > but they should be in > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/ > based on their package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4341) add types to capacity scheduler properties documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397466#comment-13397466 ] Hudson commented on MAPREDUCE-4341: --- Integrated in Hadoop-Hdfs-0.23-Build #292 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/292/]) merge -r 1351763:1351764 from branch-2. FIXES: MAPREDUCE-4341 (Revision 1351766) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351766 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm > add types to capacity scheduler properties documentation > > > Key: MAPREDUCE-4341 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4341 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched, mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Karthik Kambatla > Fix For: 0.23.3 > > Attachments: MR-4341.patch > > > MAPREDUCE-4311 is changing capacity/max capacity configuration to be floats. > We should document that in the capacity scheduler properties docs > (http://hadoop.apache.org/common/docs/r0.23.1/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Configuration). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4270) data_join test classes are in the wrong packge
[ https://issues.apache.org/jira/browse/MAPREDUCE-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397434#comment-13397434 ] Hudson commented on MAPREDUCE-4270: --- Integrated in Hadoop-Hdfs-trunk #1082 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1082/]) MAPREDUCE-4270. Move the data_join test classes to the correct path. (Contributed by Thomas Graves) (Revision 1351869) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351869 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/SampleTaggedMapOutput.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/SampleDataJoinMapper.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/SampleDataJoinReducer.java * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/SampleTaggedMapOutput.java > data_join test classes are in the wrong packge > -- > > Key: MAPREDUCE-4270 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4270 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, trunk >Reporter: Brock Noland >Assignee: Thomas Graves > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4270.sh > > > There are three Sample*.java files in this directory > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/ > but they should be in > http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-tools/hadoop-datajoin/src/test/java/org/apache/hadoop/contrib/utils/join/ > based on their package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4311) Capacity scheduler.xml does not accept decimal values for capacity and maximum-capacity settings
[ https://issues.apache.org/jira/browse/MAPREDUCE-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397435#comment-13397435 ] Hudson commented on MAPREDUCE-4311: --- Integrated in Hadoop-Hdfs-trunk #1082 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1082/]) MAPREDUCE-4311. Capacity scheduler.xml does not accept decimal values for capacity and maximum-capacity settings (Karthik Kambatla via tgraves) (Revision 1351700) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351700 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > Capacity scheduler.xml does not accept decimal values for capacity and > maximum-capacity settings > > > Key: MAPREDUCE-4311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4311 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched, mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Karthik Kambatla > Fix For: 0.23.3 > > Attachments: MR-4311.patch > > > if capacity scheduler capacity or max capacity set with decimal it errors: > - Error starting ResourceManager > java.lang.NumberFormatException: For input string: "10.5" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) > at java.lang.Integer.parseInt(Integer.java:458) > at java.lang.Integer.parseInt(Integer.java:499) > at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:713) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getCapacity(CapacitySchedulerConfiguration.java:147) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:147) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:297) > at > 0.20 used to take decimal and this could be an issue on large clusters that > would have queues with small allocations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4341) add types to capacity scheduler properties documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397433#comment-13397433 ] Hudson commented on MAPREDUCE-4341: --- Integrated in Hadoop-Hdfs-trunk #1082 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1082/]) MAPREDUCE-4341. add types to capacity scheduler properties documentation (Karthik Kambatla via tgraves). (Revision 1351762) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351762 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm > add types to capacity scheduler properties documentation > > > Key: MAPREDUCE-4341 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4341 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched, mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Karthik Kambatla > Fix For: 0.23.3 > > Attachments: MR-4341.patch > > > MAPREDUCE-4311 is changing capacity/max capacity configuration to be floats. > We should document that in the capacity scheduler properties docs > (http://hadoop.apache.org/common/docs/r0.23.1/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Configuration). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4306) Problem running Distributed Shell applications as a user other than the one started the daemons
[ https://issues.apache.org/jira/browse/MAPREDUCE-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397431#comment-13397431 ] Hudson commented on MAPREDUCE-4306: --- Integrated in Hadoop-Hdfs-trunk #1082 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1082/]) MAPREDUCE-4306. Fix distributed shell to work with users other than the one running the daemons. (Contributed by Ahmed Radwan) (Revision 1351876) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351876 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java > Problem running Distributed Shell applications as a user other than the one > started the daemons > --- > > Key: MAPREDUCE-4306 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4306 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Fix For: 2.0.1-alpha > > Attachments: MAPREDUCE-4306.patch, MAPREDUCE-4306_rev2.patch, > MAPREDUCE-4306_rev3.patch > > > Using the tarball, if you start the yarn daemons using one user and then > switch to a different user. You can successfully run MR jobs, but DS jobs > fail to run. Only able to run DS jobs using the user who started the daemons. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4267) mavenize pipes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397432#comment-13397432 ] Hudson commented on MAPREDUCE-4267: --- Integrated in Hadoop-Hdfs-trunk #1082 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1082/]) MAPREDUCE-4267. mavenize pipes (tgraves via bobby) (Revision 1351818) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1351818 Files : * /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/c++/pipes * /hadoop/common/trunk/hadoop-mapreduce-project/src/c++/utils * /hadoop/common/trunk/hadoop-mapreduce-project/src/examples/pipes * /hadoop/common/trunk/hadoop-tools/hadoop-pipes * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/.autom4te.cfg * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/Makefile.am * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/README.txt * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/configure.ac * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/.autom4te.cfg * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/Makefile.am * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/configure.ac * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/.autom4te.cfg * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/Makefile.am * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/configure.ac * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/m4/hadoop_utils.m4 * /hadoop/common/trunk/hadoop-tools/hadoop-tools-dist/pom.xml * /hadoop/common/trunk/hadoop-tools/pom.xml > mavenize pipes > -- > > Key: MAPREDUCE-4267 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4267 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3, 2.0.1-alpha, 3.0.0 > > Attachments: MAPREDUCE-4267.001.rm.patch, > MAPREDUCE-4267.001.trimmed.patch, MAPREDUCE-4267.002.trimmed.patch, > MAPREDUCE-4267.patch, MAPREDUCE-4267.patch, MAPREDUCE-4267.patch, > MAPREDUCE-4267.sh > > > We are still building pipes out of the old mrv1 directories using ant. Move > it over to the mrv2 dir structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
