[jira] [Commented] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123908#comment-13123908
]
Prashant Sharma commented on MAPREDUCE-3131:
Arun,
You are right, Adding another set of scripts should have some value more
than just starting a cluster without much hassle.
I made these scripts because it made sense for me and gave me the feel of all
possible options I can break the cluster. And I had all weird(possibly) Ideas
like trying out multiple Single node Cluster on the same machine and trying
out variety of things(like distcp).
So, In order to bring in value with a new feature. I have already implemented
following(I have not submitted the patch though) :
*To setup the cluster home in one step
{noformat}
./run.sh prepare
{noformat}
*Start stop and number of NM or Datanodes.
{noformat}
./run.sh -NM No of Nodemanager daemons -D No of datanode daemons
[start|afresh|stop|kill]
{noformat}
*Debugging and changing between or choosing between daemons for debugging would
be breeze.
{noformat}
./run.sh -NM port no -RM port no -DN port no -NN portno -JH port no
debug
{noformat}
I will soon submit the patch with much better code quality. I understand the
hassle of maintaining a set of badly written code (Specially scripts). Also I
have already written some default configs ideal for single node cluster. I
liked the naming conventions.
Now do you see some value.? I would like to have you people throwing in ideas.
:)
Thanks
Docs and Scripts for setting up single node MRV2 cluster.
--
Key: MAPREDUCE-3131
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: documentation, mrv2
Affects Versions: 0.24.0
Reporter: Prashant Sharma
Assignee: Prashant Sharma
Priority: Trivial
Labels: documentation, hadoop
Fix For: 0.24.0
Attachments: MAPREDUCE-3131.patch, MAPREDUCE-3131.patch
Original Estimate: 168h
Time Spent: 48h
Remaining Estimate: 120h
Scripts to run a single node cluster with a default configuration. Takes care
of running all the daemons including hdfs and yarn.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123911#comment-13123911
]
Prashant Sharma commented on MAPREDUCE-3131:
Edit in step 2
{noformat}
./run.sh -NM No of Nodemanager daemons -D No of datanode daemons
[start|afresh|stop|kill|status]
{noformat}
Added status option. Currently work in progress. Status will check pid files of
process and querry system about the status of those processes. Would be useful
in case you run multiple clusters.
Docs and Scripts for setting up single node MRV2 cluster.
--
Key: MAPREDUCE-3131
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: documentation, mrv2
Affects Versions: 0.24.0
Reporter: Prashant Sharma
Assignee: Prashant Sharma
Priority: Trivial
Labels: documentation, hadoop
Fix For: 0.24.0
Attachments: MAPREDUCE-3131.patch, MAPREDUCE-3131.patch
Original Estimate: 168h
Time Spent: 48h
Remaining Estimate: 120h
Scripts to run a single node cluster with a default configuration. Takes care
of running all the daemons including hdfs and yarn.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2988: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2988: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Hadoop Flags: Reviewed Status: Patch Available (was: Open) Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2988: --- Attachment: MAPREDUCE-2988-20111010.txt Latest patch looks good. Trivial update to add the missing configure goal. Without this, fresh checkout fails to compile the binary. Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123946#comment-13123946 ] Hadoop QA commented on MAPREDUCE-2988: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498395/MAPREDUCE-2988-20111010.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/976//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/976//console This message is automatically generated. Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3104) Implement Application ACLs, Queue ACLs and their interaction
[ https://issues.apache.org/jira/browse/MAPREDUCE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reassigned MAPREDUCE-3104: -- Assignee: Vinod Kumar Vavilapalli Implement Application ACLs, Queue ACLs and their interaction Key: MAPREDUCE-3104 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3104 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-2988: --- Resolution: Fixed Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Resolved (was: Patch Available) I just committed this to trunk and branch-0.23. Thanks Robert! Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3104) Implement Application ACLs, Queue ACLs and their interaction
[ https://issues.apache.org/jira/browse/MAPREDUCE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3104: --- Component/s: mrv2 Implement Application ACLs, Queue ACLs and their interaction Key: MAPREDUCE-3104 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3104 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3104) Implement Application ACLs, Queue ACLs and their interaction
[ https://issues.apache.org/jira/browse/MAPREDUCE-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3104: --- Component/s: resourcemanager Priority: Blocker (was: Major) Affects Version/s: 0.23.0 Implement Application ACLs, Queue ACLs and their interaction Key: MAPREDUCE-3104 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3104 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, resourcemanager, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123972#comment-13123972 ] Hudson commented on MAPREDUCE-2988: --- Integrated in Hadoop-Common-trunk-Commit #1050 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1050/]) MAPREDUCE-2988. Reenabled TestLinuxContainerExecutor reflecting the current NodeManager code. Contributed by Robert Joseph Evans. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180833 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/.gitignore * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/Makefile.am * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/configure.ac * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-task-controller.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123973#comment-13123973 ] Hudson commented on MAPREDUCE-2988: --- Integrated in Hadoop-Hdfs-trunk-Commit #1128 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1128/]) MAPREDUCE-2988. Reenabled TestLinuxContainerExecutor reflecting the current NodeManager code. Contributed by Robert Joseph Evans. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180833 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/.gitignore * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/Makefile.am * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/configure.ac * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-task-controller.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2795) [MR-279] AppsKilled is never incremented
[
https://issues.apache.org/jira/browse/MAPREDUCE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13123979#comment-13123979
]
Devaraj K commented on MAPREDUCE-2795:
--
Hi Vinod,
Thanks for the explanation.
{code:title=YARNRunner.java|borderStyle=solid}
public void killJob(JobID arg0) throws IOException, InterruptedException {
/* check if the status is not running, if not send kill to RM */
JobStatus status = clientCache.getClient(arg0).getJobStatus(arg0);
if (status.getState() != JobStatus.State.RUNNING) {
resMgrDelegate.killApplication(TypeConverter.toYarn(arg0).getAppId());
return;
}
try {
/* send a kill to the AM */
clientCache.getClient(arg0).killJob(arg0);
long currentTimeMillis = System.currentTimeMillis();
long timeKillIssued = currentTimeMillis;
while ((currentTimeMillis timeKillIssued + 1L) (status.getState()
!= JobStatus.State.KILLED)) {
try {
Thread.sleep(1000L);
} catch(InterruptedException ie) {
/** interrupted, just break */
break;
}
currentTimeMillis = System.currentTimeMillis();
status = clientCache.getClient(arg0).getJobStatus(arg0);
}
} catch(IOException io) {
LOG.debug(Error when checking for application status, io);
}
if (status.getState() != JobStatus.State.KILLED) {
resMgrDelegate.killApplication(TypeConverter.toYarn(arg0).getAppId());
}
}
{code}
If I refer to YARNRunner.killJob(JobID) method, then I see that the killing of
the job is handled in two different ways.
1. If JobStatus.State. is not RUNNING then
ResourceMgrDelegate.killApplication(ApplicationId)
2. Otherwise ClientServiceDelegate.killJob(JobID)
Is it good idea to kill the application itself, if it not in running state? In
this case it is incrementing the apps killed metrics even if the user trying to
kill the job.
[MR-279] AppsKilled is never incremented
Key: MAPREDUCE-2795
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2795
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Devaraj K
Priority: Blocker
Fix For: 0.23.0, 0.24.0
Attachments: MAPREDUCE-2795.patch
AppsKilled metrics is never incremented even though there are killed jobs in
the system.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124018#comment-13124018 ] Hudson commented on MAPREDUCE-3138: --- Integrated in Hadoop-Hdfs-trunk #826 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/]) Move MAPREDUCE-3138 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180751 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Allow for applications to deal with MAPREDUCE-954 - Key: MAPREDUCE-3138 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Owen O'Malley Priority: Blocker Fix For: 0.22.0, 0.23.0 Attachments: MAPREDUCE-3138.0.22.patch, MAPREDUCE-3138.patch, MAPREDUCE-3138.patch MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124020#comment-13124020 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Hdfs-trunk #826 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/]) MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by MAPREDUCE-2702. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180769 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124021#comment-13124021 ] Hudson commented on MAPREDUCE-2876: --- Integrated in Hadoop-Hdfs-trunk #826 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/]) MAPREDUCE-2876. Use a different config for ContainerAllocationExpirer. Contributed by Anupam Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180767 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NMLivelinessMonitor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/AMLivelinessMonitor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/ContainerAllocationExpirer.java ContainerAllocationExpirer appears to use the incorrect configs --- Key: MAPREDUCE-2876 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Anupam Seth Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2876-branch-0_23.patch, MAPREDUCE-2876-branch-0_23.patch ContainerAllocationExpirer sets the expiration interval to be RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different from what AMLivelinessMonitor does. There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124024#comment-13124024 ] Hudson commented on MAPREDUCE-3153: --- Integrated in Hadoop-Hdfs-trunk #826 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/]) MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by MAPREDUCE-2702. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180769 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins -- Key: MAPREDUCE-3153 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Mahadev konar Fix For: 0.23.0 Attachments: MAPREDUCE-3153.patch This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124023#comment-13124023 ] Hudson commented on MAPREDUCE-3123: --- Integrated in Hadoop-Hdfs-trunk #826 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/]) MAPREDUCE-3123. Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180772 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Symbolic links with special chars causing container/task.sh to fail --- Key: MAPREDUCE-3123 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3123.1.patch, MR-3123.2.patch, MR-3123.3.patch the following job throws an exception when you have the special characters in it. hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* -Dmapreduce.job.queuename=queue1 -files file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$*()-_+= -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat -output Streaming/streaming-980/Output -jobconf mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* Exception: 2011-09-27 20:58:48,903 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: launchContainer: [container-executor, hadoopuser, 1, application_1317077272567_0239, container_1317077272567 0239_01_01, tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 01/task.sh, tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 20:58:48,944 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container is : 2 2011-09-27 20:58:48,946 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception from container-launch : org.apache.hadoop.util.Shell$ExitCodeException: /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: line 26: syntax error near unexpected token `-_+=' /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: line 26: `ln -sf /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir test ink!@$*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at org.apache.hadoop.util.Shell.run(Shell.java:188) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174)
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124025#comment-13124025 ] Hudson commented on MAPREDUCE-2988: --- Integrated in Hadoop-Hdfs-trunk #826 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/]) MAPREDUCE-2988. Reenabled TestLinuxContainerExecutor reflecting the current NodeManager code. Contributed by Robert Joseph Evans. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180833 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/.gitignore * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/Makefile.am * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/configure.ac * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-task-controller.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124048#comment-13124048 ] Hudson commented on MAPREDUCE-3138: --- Integrated in Hadoop-Hdfs-0.23-Build #35 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/]) Move MAPREDUCE-3138 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180753 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt Allow for applications to deal with MAPREDUCE-954 - Key: MAPREDUCE-3138 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Owen O'Malley Priority: Blocker Fix For: 0.22.0, 0.23.0 Attachments: MAPREDUCE-3138.0.22.patch, MAPREDUCE-3138.patch, MAPREDUCE-3138.patch MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3154) Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method
[
https://issues.apache.org/jira/browse/MAPREDUCE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124049#comment-13124049
]
Hudson commented on MAPREDUCE-3154:
---
Integrated in Hadoop-Hdfs-0.23-Build #35 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/])
Merge -c 1180774 from trunk to branch-0.23 to fix MAPREDUCE-3154.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180775
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
Validate the Jobs Output Specification as the first statement in
JobSubmitter.submitJobInternal(Job, Cluster) method
Key: MAPREDUCE-3154
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3154
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client, mrv2
Affects Versions: 0.23.0, 0.24.0
Environment: mrv2
Reporter: Abhijit Suresh Shingate
Assignee: Abhijit Suresh Shingate
Fix For: 0.23.0
Attachments: MAPREDUCE-3154.patch
Original Estimate: 1h
Remaining Estimate: 1h
Presently the output specification is validated after getting new JobId from
ClientRMService, Copying the job jar, Configuration file, archives etc.
Instead of that move following Job Output specification validation call to
the begining of JobSubmitter.submitJobInternal(Job, Cluster) method.
{code}
checkSpecs(job);
{code}
This will avoid unnecessary work in case of invalid output specs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124050#comment-13124050 ] Hudson commented on MAPREDUCE-2876: --- Integrated in Hadoop-Hdfs-0.23-Build #35 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/]) Merge -c 1180767 from trunk to branch-0.23 to fix MAPREDUCE-2876. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180768 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NMLivelinessMonitor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/AMLivelinessMonitor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/ContainerAllocationExpirer.java ContainerAllocationExpirer appears to use the incorrect configs --- Key: MAPREDUCE-2876 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Anupam Seth Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2876-branch-0_23.patch, MAPREDUCE-2876-branch-0_23.patch ContainerAllocationExpirer sets the expiration interval to be RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different from what AMLivelinessMonitor does. There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124052#comment-13124052 ] Hudson commented on MAPREDUCE-3123: --- Integrated in Hadoop-Hdfs-0.23-Build #35 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/]) Merge -c 1180772 from trunk to branch-0.23 to fix MAPREDUCE-3123. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180773 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Symbolic links with special chars causing container/task.sh to fail --- Key: MAPREDUCE-3123 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3123.1.patch, MR-3123.2.patch, MR-3123.3.patch the following job throws an exception when you have the special characters in it. hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* -Dmapreduce.job.queuename=queue1 -files file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$*()-_+= -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat -output Streaming/streaming-980/Output -jobconf mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* Exception: 2011-09-27 20:58:48,903 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: launchContainer: [container-executor, hadoopuser, 1, application_1317077272567_0239, container_1317077272567 0239_01_01, tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 01/task.sh, tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 20:58:48,944 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container is : 2 2011-09-27 20:58:48,946 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception from container-launch : org.apache.hadoop.util.Shell$ExitCodeException: /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: line 26: syntax error near unexpected token `-_+=' /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: line 26: `ln -sf /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir test ink!@$*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at org.apache.hadoop.util.Shell.run(Shell.java:188) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174)
[jira] [Commented] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124053#comment-13124053 ] Hudson commented on MAPREDUCE-3153: --- Integrated in Hadoop-Hdfs-0.23-Build #35 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/]) Merge -c 1180769 from trunk to branch-0.23 to fix MAPREDUCE-3153. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180770 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins -- Key: MAPREDUCE-3153 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Mahadev konar Fix For: 0.23.0 Attachments: MAPREDUCE-3153.patch This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124054#comment-13124054 ] Hudson commented on MAPREDUCE-2988: --- Integrated in Hadoop-Hdfs-0.23-Build #35 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/]) MAPREDUCE-2988. svn merge -c r1180833 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180835 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/Makefile.am * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/configure.ac * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-task-controller.c * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2795) [MR-279] AppsKilled is never incremented
[ https://issues.apache.org/jira/browse/MAPREDUCE-2795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124055#comment-13124055 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2795: That is by design. In case of mapreduce running over YARN, as of now, one mapreduce job is mapped to one YARN application. So, if the application has not yet started running, we kill the application itself so as to not start the job (the AM precisely). In which case, YARN duely increments the killedApps metric, correctly. [MR-279] AppsKilled is never incremented Key: MAPREDUCE-2795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2795 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Devaraj K Priority: Blocker Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2795.patch AppsKilled metrics is never incremented even though there are killed jobs in the system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-954) The new interface's Context objects should be interfaces
[ https://issues.apache.org/jira/browse/MAPREDUCE-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124066#comment-13124066 ] Hudson commented on MAPREDUCE-954: -- Integrated in Hadoop-Mapreduce-22-branch #83 (See [https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/83/]) MAPREDUCE-3138. Add a utility to help applications bridge changes in Context Objects APIs due to MAPREDUCE-954. Contributed by omalley. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180750 Files : * /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt * /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapreduce/ContextFactory.java The new interface's Context objects should be interfaces Key: MAPREDUCE-954 URL: https://issues.apache.org/jira/browse/MAPREDUCE-954 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Owen O'Malley Assignee: Arun C Murthy Fix For: 0.21.0 Attachments: MAPREDUCE-954.patch, MAPREDUCE-954.patch, MAPREDUCE-954.patch, MAPREDUCE-954.patch, MAPREDUCE-954.patch, MAPREDUCE-954.patch, MAPREDUCE-954.patch When I was doing HADOOP-1230, I was persuaded to make the Context objects as classes. I think that was a serious mistake. It caused a lot of information leakage into the public classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124065#comment-13124065 ] Hudson commented on MAPREDUCE-3138: --- Integrated in Hadoop-Mapreduce-22-branch #83 (See [https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/83/]) MAPREDUCE-3138. Add a utility to help applications bridge changes in Context Objects APIs due to MAPREDUCE-954. Contributed by omalley. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180750 Files : * /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt * /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapreduce/ContextFactory.java Allow for applications to deal with MAPREDUCE-954 - Key: MAPREDUCE-3138 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Owen O'Malley Priority: Blocker Fix For: 0.22.0, 0.23.0 Attachments: MAPREDUCE-3138.0.22.patch, MAPREDUCE-3138.patch, MAPREDUCE-3138.patch MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3154) Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method
[
https://issues.apache.org/jira/browse/MAPREDUCE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124074#comment-13124074
]
Hudson commented on MAPREDUCE-3154:
---
Integrated in Hadoop-Mapreduce-0.23-Build #42 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/])
Merge -c 1180774 from trunk to branch-0.23 to fix MAPREDUCE-3154.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180775
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
Validate the Jobs Output Specification as the first statement in
JobSubmitter.submitJobInternal(Job, Cluster) method
Key: MAPREDUCE-3154
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3154
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client, mrv2
Affects Versions: 0.23.0, 0.24.0
Environment: mrv2
Reporter: Abhijit Suresh Shingate
Assignee: Abhijit Suresh Shingate
Fix For: 0.23.0
Attachments: MAPREDUCE-3154.patch
Original Estimate: 1h
Remaining Estimate: 1h
Presently the output specification is validated after getting new JobId from
ClientRMService, Copying the job jar, Configuration file, archives etc.
Instead of that move following Job Output specification validation call to
the begining of JobSubmitter.submitJobInternal(Job, Cluster) method.
{code}
checkSpecs(job);
{code}
This will avoid unnecessary work in case of invalid output specs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124073#comment-13124073 ] Hudson commented on MAPREDUCE-3138: --- Integrated in Hadoop-Mapreduce-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/]) Move MAPREDUCE-3138 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180753 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt Allow for applications to deal with MAPREDUCE-954 - Key: MAPREDUCE-3138 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Owen O'Malley Priority: Blocker Fix For: 0.22.0, 0.23.0 Attachments: MAPREDUCE-3138.0.22.patch, MAPREDUCE-3138.patch, MAPREDUCE-3138.patch MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124075#comment-13124075 ] Hudson commented on MAPREDUCE-2876: --- Integrated in Hadoop-Mapreduce-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/]) Merge -c 1180767 from trunk to branch-0.23 to fix MAPREDUCE-2876. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180768 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NMLivelinessMonitor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/AMLivelinessMonitor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/ContainerAllocationExpirer.java ContainerAllocationExpirer appears to use the incorrect configs --- Key: MAPREDUCE-2876 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Anupam Seth Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2876-branch-0_23.patch, MAPREDUCE-2876-branch-0_23.patch ContainerAllocationExpirer sets the expiration interval to be RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different from what AMLivelinessMonitor does. There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124077#comment-13124077 ] Hudson commented on MAPREDUCE-3123: --- Integrated in Hadoop-Mapreduce-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/]) Merge -c 1180772 from trunk to branch-0.23 to fix MAPREDUCE-3123. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180773 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Symbolic links with special chars causing container/task.sh to fail --- Key: MAPREDUCE-3123 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3123.1.patch, MR-3123.2.patch, MR-3123.3.patch the following job throws an exception when you have the special characters in it. hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* -Dmapreduce.job.queuename=queue1 -files file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$*()-_+= -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat -output Streaming/streaming-980/Output -jobconf mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* Exception: 2011-09-27 20:58:48,903 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: launchContainer: [container-executor, hadoopuser, 1, application_1317077272567_0239, container_1317077272567 0239_01_01, tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 01/task.sh, tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 20:58:48,944 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container is : 2 2011-09-27 20:58:48,946 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception from container-launch : org.apache.hadoop.util.Shell$ExitCodeException: /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: line 26: syntax error near unexpected token `-_+=' /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: line 26: `ln -sf /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir test ink!@$*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at org.apache.hadoop.util.Shell.run(Shell.java:188) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174)
[jira] [Commented] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124078#comment-13124078 ] Hudson commented on MAPREDUCE-3153: --- Integrated in Hadoop-Mapreduce-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/]) Merge -c 1180769 from trunk to branch-0.23 to fix MAPREDUCE-3153. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180770 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins -- Key: MAPREDUCE-3153 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Mahadev konar Fix For: 0.23.0 Attachments: MAPREDUCE-3153.patch This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124079#comment-13124079 ] Hudson commented on MAPREDUCE-2988: --- Integrated in Hadoop-Mapreduce-0.23-Build #42 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/]) MAPREDUCE-2988. svn merge -c r1180833 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180835 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/Makefile.am * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/configure.ac * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-task-controller.c * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124082#comment-13124082 ] Hudson commented on MAPREDUCE-3138: --- Integrated in Hadoop-Mapreduce-trunk #856 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/]) Move MAPREDUCE-3138 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180751 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Allow for applications to deal with MAPREDUCE-954 - Key: MAPREDUCE-3138 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Owen O'Malley Priority: Blocker Fix For: 0.22.0, 0.23.0 Attachments: MAPREDUCE-3138.0.22.patch, MAPREDUCE-3138.patch, MAPREDUCE-3138.patch MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124085#comment-13124085 ] Hudson commented on MAPREDUCE-2876: --- Integrated in Hadoop-Mapreduce-trunk #856 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/]) MAPREDUCE-2876. Use a different config for ContainerAllocationExpirer. Contributed by Anupam Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180767 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NMLivelinessMonitor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/AMLivelinessMonitor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/ContainerAllocationExpirer.java ContainerAllocationExpirer appears to use the incorrect configs --- Key: MAPREDUCE-2876 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Anupam Seth Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2876-branch-0_23.patch, MAPREDUCE-2876-branch-0_23.patch ContainerAllocationExpirer sets the expiration interval to be RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different from what AMLivelinessMonitor does. There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3154) Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method
[
https://issues.apache.org/jira/browse/MAPREDUCE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124083#comment-13124083
]
Hudson commented on MAPREDUCE-3154:
---
Integrated in Hadoop-Mapreduce-trunk #856 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/])
MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying
job submission files to fail fast. Contributed by Abhijit Suresh Shingate.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180774
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
Validate the Jobs Output Specification as the first statement in
JobSubmitter.submitJobInternal(Job, Cluster) method
Key: MAPREDUCE-3154
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3154
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client, mrv2
Affects Versions: 0.23.0, 0.24.0
Environment: mrv2
Reporter: Abhijit Suresh Shingate
Assignee: Abhijit Suresh Shingate
Fix For: 0.23.0
Attachments: MAPREDUCE-3154.patch
Original Estimate: 1h
Remaining Estimate: 1h
Presently the output specification is validated after getting new JobId from
ClientRMService, Copying the job jar, Configuration file, archives etc.
Instead of that move following Job Output specification validation call to
the begining of JobSubmitter.submitJobInternal(Job, Cluster) method.
{code}
checkSpecs(job);
{code}
This will avoid unnecessary work in case of invalid output specs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124084#comment-13124084 ] Hudson commented on MAPREDUCE-2702: --- Integrated in Hadoop-Mapreduce-trunk #856 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/]) MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by MAPREDUCE-2702. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180769 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java [MR-279] OutputCommitter changes for MR Application Master recovery --- Key: MAPREDUCE-2702 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Sharad Agarwal Assignee: Sharad Agarwal Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch In MR AM recovers from a crash, it only reruns the non completed tasks. The completed tasks (along with their output, if any) needs to be recovered from the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124088#comment-13124088 ] Hudson commented on MAPREDUCE-3153: --- Integrated in Hadoop-Mapreduce-trunk #856 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/]) MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by MAPREDUCE-2702. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180769 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins -- Key: MAPREDUCE-3153 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Mahadev konar Fix For: 0.23.0 Attachments: MAPREDUCE-3153.patch This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124087#comment-13124087 ] Hudson commented on MAPREDUCE-3123: --- Integrated in Hadoop-Mapreduce-trunk #856 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/]) MAPREDUCE-3123. Fix NM to quote symlink names to escape special characters. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180772 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java Symbolic links with special chars causing container/task.sh to fail --- Key: MAPREDUCE-3123 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.0 Attachments: MR-3123.1.patch, MR-3123.2.patch, MR-3123.3.patch the following job throws an exception when you have the special characters in it. hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* -Dmapreduce.job.queuename=queue1 -files file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$*()-_+= -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat -output Streaming/streaming-980/Output -jobconf mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* Exception: 2011-09-27 20:58:48,903 INFO org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: launchContainer: [container-executor, hadoopuser, 1, application_1317077272567_0239, container_1317077272567 0239_01_01, tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 01/task.sh, tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 20:58:48,944 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container is : 2 2011-09-27 20:58:48,946 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception from container-launch : org.apache.hadoop.util.Shell$ExitCodeException: /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: line 26: syntax error near unexpected token `-_+=' /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: line 26: `ln -sf /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir test ink!@$*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) at org.apache.hadoop.util.Shell.run(Shell.java:188) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174)
[jira] [Commented] (MAPREDUCE-2988) Reenable TestLinuxContainerExecutor reflecting the current NM code.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124089#comment-13124089 ] Hudson commented on MAPREDUCE-2988: --- Integrated in Hadoop-Mapreduce-trunk #856 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/]) MAPREDUCE-2988. Reenabled TestLinuxContainerExecutor reflecting the current NodeManager code. Contributed by Robert Joseph Evans. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1180833 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/.gitignore * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/Makefile.am * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/configure.ac * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-task-controller.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor Reenable TestLinuxContainerExecutor reflecting the current NM code. Key: MAPREDUCE-2988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2988 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security, test Affects Versions: 0.23.0, 0.24.0 Reporter: Eric Payne Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2988-20111010.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt, MR-2988.txt Original Estimate: 72h Remaining Estimate: 72h TestLinuxContainerExecutor is currently disabled completely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3152) Miscellaneous web UI issues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124098#comment-13124098 ] Subroto Sanyal commented on MAPREDUCE-3152: --- Hi Vinod, bq. When the Application is still not running, the Tracking UI should be title UNASSIGNED, for some reason it is titled ApplicationMaster but (correctly) links to #. This point has been taken care in MAPREDUCE-3140 Miscellaneous web UI issues --- Key: MAPREDUCE-3152 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3152 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.0 We need to fix the following issues on YARN web-UI: - Remove the Note column from the application list. When a failure happens, this Note spoils the table layout. - When the Application is still not running, the Tracking UI should be title UNASSIGNED, for some reason it is titled ApplicationMaster but (correctly) links to #. - The per-application page has all the RM related information like version, start-time etc. Must be some accidental change by one of the patches. - The diagnostics for a failed app on the per-application page don't retain new lines and wrap'em around - looks hard to read. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3157) Rumen TraceBuilder is skipping analyzing 0.20 history files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3157: Attachment: MR3157.patch Attaching patch fixing the issue by adding 2 regular expressions for Pre21 history file name formats and 2 regular expressions for Pre21 history conf file name formats. Also did some refactoring of code. Updated TestRumenJobTraces to check all the 3 history file name formats and the 3 history conf file name formats. Rumen TraceBuilder is skipping analyzing 0.20 history files --- Key: MAPREDUCE-3157 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3157 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: MR3157.patch Rumen TraceBuilder is assuming the Pre21 history file name format to be JTIdentifier_jobId_something. But it can be jobId_something also as it is now in latest 0.20.x version. This also needs to be understood by TraceBuilder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3157) Rumen TraceBuilder is skipping analyzing 0.20 history files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3157: Status: Patch Available (was: Open) Rumen TraceBuilder is skipping analyzing 0.20 history files --- Key: MAPREDUCE-3157 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3157 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: MR3157.patch Rumen TraceBuilder is assuming the Pre21 history file name format to be JTIdentifier_jobId_something. But it can be jobId_something also as it is now in latest 0.20.x version. This also needs to be understood by TraceBuilder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124123#comment-13124123 ] Anupam Seth commented on MAPREDUCE-2876: Thanks Bobby, Mahadev and Arun! ContainerAllocationExpirer appears to use the incorrect configs --- Key: MAPREDUCE-2876 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Anupam Seth Priority: Critical Fix For: 0.23.0 Attachments: MAPREDUCE-2876-branch-0_23.patch, MAPREDUCE-2876-branch-0_23.patch ContainerAllocationExpirer sets the expiration interval to be RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different from what AMLivelinessMonitor does. There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3157) Rumen TraceBuilder is skipping analyzing 0.20 history files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124128#comment-13124128 ] Hadoop QA commented on MAPREDUCE-3157: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498427/MR3157.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/977//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/977//console This message is automatically generated. Rumen TraceBuilder is skipping analyzing 0.20 history files --- Key: MAPREDUCE-3157 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3157 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: MR3157.patch Rumen TraceBuilder is assuming the Pre21 history file name format to be JTIdentifier_jobId_something. But it can be jobId_something also as it is now in latest 0.20.x version. This also needs to be understood by TraceBuilder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1688) A failing retry'able notification in JobEndNotifier can affect notifications of other jobs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124171#comment-13124171
]
Ravi Prakash commented on MAPREDUCE-1688:
-
@Vinod : Can't we set the maximum amount of time an HttpMethod can wait when we
create it in JobEndNotifier.httpNotification() ? Perhaps using the parameters
defined in http://hc.apache.org/httpclient-3.x/preference-api.html ?
A failing retry'able notification in JobEndNotifier can affect notifications
of other jobs.
---
Key: MAPREDUCE-1688
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1688
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.20.1
Reporter: Vinod Kumar Vavilapalli
The JobTracker puts all the notification commands into a delay-queue. It has
a single thread that loops through this queue and sends out the
notifications. When it hits failures with any notification which is
configured to be retired via {{job.end.retry.attempts}} and
{{job.end.retry.interval}}, the notification is queued back again. A single
notification with sufficiently large number of configured retries and which
consistently fails will affect other notifications in the queue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2668) MR-279: APPLICATION_STOP is never sent to AuxServices
[ https://issues.apache.org/jira/browse/MAPREDUCE-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124319#comment-13124319 ] jirapos...@reviews.apache.org commented on MAPREDUCE-2668: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2332/ --- Review request for Robert Evans and Jonathan Eagles. Summary --- APPLICATION_STOP is never sent to the AuxServices only APPLICATION_INIT. This addresses bug MAPREDUCE-2668. https://issues.apache.org/jira/browse/MAPREDUCE-2668 Diffs - http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java 1180995 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java 1180995 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java 1180995 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestAuxServices.java 1180995 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java 1180995 Diff: https://reviews.apache.org/r/2332/diff Testing --- Thanks, Tom MR-279: APPLICATION_STOP is never sent to AuxServices - Key: MAPREDUCE-2668 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2668 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2668.patch, MAPREDUCE-2668.patch APPLICATION_STOP is never sent to the AuxServices only APPLICATION_INIT. This means that all map intermediate data will never be deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2668) MR-279: APPLICATION_STOP is never sent to AuxServices
[ https://issues.apache.org/jira/browse/MAPREDUCE-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2668: - Attachment: MAPREDUCE-2668.patch MR-279: APPLICATION_STOP is never sent to AuxServices - Key: MAPREDUCE-2668 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2668 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2668.patch, MAPREDUCE-2668.patch APPLICATION_STOP is never sent to the AuxServices only APPLICATION_INIT. This means that all map intermediate data will never be deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2668) MR-279: APPLICATION_STOP is never sent to AuxServices
[ https://issues.apache.org/jira/browse/MAPREDUCE-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2668: - Status: Patch Available (was: Open) MR-279: APPLICATION_STOP is never sent to AuxServices - Key: MAPREDUCE-2668 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2668 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2668.patch, MAPREDUCE-2668.patch APPLICATION_STOP is never sent to the AuxServices only APPLICATION_INIT. This means that all map intermediate data will never be deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah reopened MAPREDUCE-3033:
Re-opening issue to address MRv1 commit build test failures
JobClient requires mapreduce.jobtracker.address config even when
mapreduce.framework.name is set to yarn
Key: MAPREDUCE-3033
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission, mrv2
Affects Versions: 0.23.0
Reporter: Karam Singh
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.0
Attachments: MR-3033.1.patch, MR-3033.2.patch
If mapreduce.jobtracker.address is not set in mapred-site.xml and
mapreduce.framework.name is set yarn, job submission fails :
Tried to submit sleep job with maps 1 task. Job submission failed with
following exception -:
{code}
11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
ResourceManager at RMHost:8040
11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
ResourceManager at RMHost:8040
11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
getStagingAreaDir: dir=/user/username/.staging
11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
/user/username/.staging/job_1316435926198_0004
java.lang.RuntimeException: Not a host:port pair: local
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at
org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
at
org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1849) Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124347#comment-13124347 ] Jeff Hammerbacher commented on MAPREDUCE-1849: -- Josh Wills from Cloudera has implemented the FlumeJava API in an open source project called Crunch: http://www.cloudera.com/blog/2011/10/introducing-crunch/. The code is currently on Github: https://github.com/cloudera/crunch. Once we've built a community of contributors, we plan to take the project to the Apache Incubator. If you're interested in programmatic workflow construction from Java, we'd welcome your contributions! Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce -- Key: MAPREDUCE-1849 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Jeff Hammerbacher The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1849) Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124346#comment-13124346 ] Todd Lipcon commented on MAPREDUCE-1849: Josh Wills has implemented this as a project on github: https://github.com/cloudera/crunch/ Implement a FlumeJava-like library for operations over parallel collections using Hadoop MapReduce -- Key: MAPREDUCE-1849 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1849 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Jeff Hammerbacher The API used internally at Google is described in great detail at http://portal.acm.org/citation.cfm?id=1806596.1806638. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2668) MR-279: APPLICATION_STOP is never sent to AuxServices
[ https://issues.apache.org/jira/browse/MAPREDUCE-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124354#comment-13124354 ] Hadoop QA commented on MAPREDUCE-2668: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498444/MAPREDUCE-2668.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/978//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/978//console This message is automatically generated. MR-279: APPLICATION_STOP is never sent to AuxServices - Key: MAPREDUCE-2668 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2668 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.0 Attachments: MAPREDUCE-2668.patch, MAPREDUCE-2668.patch APPLICATION_STOP is never sent to the AuxServices only APPLICATION_INIT. This means that all map intermediate data will never be deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah resolved MAPREDUCE-3033.
Resolution: Fixed
Will open a new jira for the mrv1 test failures.
JobClient requires mapreduce.jobtracker.address config even when
mapreduce.framework.name is set to yarn
Key: MAPREDUCE-3033
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission, mrv2
Affects Versions: 0.23.0
Reporter: Karam Singh
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.0
Attachments: MR-3033.1.patch, MR-3033.2.patch
If mapreduce.jobtracker.address is not set in mapred-site.xml and
mapreduce.framework.name is set yarn, job submission fails :
Tried to submit sleep job with maps 1 task. Job submission failed with
following exception -:
{code}
11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
ResourceManager at RMHost:8040
11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
ResourceManager at RMHost:8040
11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
getStagingAreaDir: dir=/user/username/.staging
11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
/user/username/.staging/job_1316435926198_0004
java.lang.RuntimeException: Not a host:port pair: local
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
at
org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at
org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
at
org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3158) Fix trunk build failures
Fix trunk build failures Key: MAPREDUCE-3158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3159) DefaultContainerExecutor removes appcache dir on every localization
DefaultContainerExecutor removes appcache dir on every localization --- Key: MAPREDUCE-3159 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3159 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker The DefaultContainerExecutor currently has code that removes the application dir from appcache/ in the local directories on every task localization. This causes any concurrent executing tasks from the same job to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124473#comment-13124473 ] Jonathan Eagles commented on MAPREDUCE-2666: Accepting this ticket after talking with Bobby. MR-279: Need to retrieve shuffle port number on ApplicationMaster restart - Key: MAPREDUCE-2666 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Priority: Blocker Fix For: 0.23.0 MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on. In the case of an ApplicationMaster crash where it needs to be restarted that information is lost. We either need to re-query it from each of the NodeManagers or to persist it to the JobHistory logs and retrieve it again. The job history logs is probably the simpler solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned MAPREDUCE-2666: -- Assignee: Jonathan Eagles (was: Robert Joseph Evans) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart - Key: MAPREDUCE-2666 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.0 MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on. In the case of an ApplicationMaster crash where it needs to be restarted that information is lost. We either need to re-query it from each of the NodeManagers or to persist it to the JobHistory logs and retrieve it again. The job history logs is probably the simpler solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3160) Merge -r 1177530:1177531 from trunk to branch-0.23 to fix MAPREDUCE-2996 broke ant test compilation
Merge -r 1177530:1177531 from trunk to branch-0.23 to fix MAPREDUCE-2996 broke
ant test compilation
---
Key: MAPREDUCE-3160
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3160
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ravi Prakash
I git bisected and the problem starts from
adb810babaf25b9f9dae75b43d4beac782deaa01 . ant
{noformat}
[jsp-compile] log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[javac]
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/build.xml:398:
warning: 'includeantruntime' was not set, defaulting to
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 2 source files to
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/build/classes
[javac]
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java:697:
cannot find symbol
[javac] symbol : constructor
JobInitedEvent(org.apache.hadoop.mapred.JobID,long,int,int,java.lang.String,boolean)
[javac] location: class
org.apache.hadoop.mapreduce.jobhistory.JobInitedEvent
[javac] JobInitedEvent jie = new JobInitedEvent(
[javac] ^
[javac] Note:
/home/raviprak/Code/hadoop/hadoop-all/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java
uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 1 error
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3162) Separate application-init and container-init event types in NM's ApplicationImpl FSM
Separate application-init and container-init event types in NM's ApplicationImpl FSM Key: MAPREDUCE-3162 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-3161: --- Attachment: mr-3161.txt Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-2789: -- Attachment: MAPREDUCE-2789.v2.txt @Arun, I have addressed the previous comments. Please see new patch. [MR:279] Update the scheduling info on CLI -- Key: MAPREDUCE-2789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Eric Payne Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2789.v1.txt, MAPREDUCE-2789.v2.txt The scheduling information such as number of containers running, memory usage and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3158) Fix trunk build failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3158: --- Status: Patch Available (was: Open) Fix trunk build failures Key: MAPREDUCE-3158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 Attachments: MR-3158.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3158) Fix trunk build failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3158: --- Attachment: MR-3158.1.patch Changed ReduceTask to assume a local job iff framework name is set to local ( removed check on job tracker being local ). Also, modified LocalContainerLauncher to set framework name to local to be able to run local reduce tasks. Changes to a lot of tests which were failing/timing out due to the framework name change. Most tests now modified to use framework = local which allows them to work normally. All mvn tests and ant commit tests work now. All smoke tests except TestMiniMRChildTask work. Errors in that test seem to be due to jetty failures. Fix trunk build failures Key: MAPREDUCE-3158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 Attachments: MR-3158.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-3161: --- Status: Patch Available (was: Open) Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3158) Fix trunk build failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124594#comment-13124594 ] Hadoop QA commented on MAPREDUCE-3158: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498489/MR-3158.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 34 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/979//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/979//console This message is automatically generated. Fix trunk build failures Key: MAPREDUCE-3158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 Attachments: MR-3158.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3158) Fix trunk build failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3158: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! Fix trunk build failures Key: MAPREDUCE-3158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 Attachments: MR-3158.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124600#comment-13124600 ] Hadoop QA commented on MAPREDUCE-3161: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498481/mr-3161.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/980//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/980//console This message is automatically generated. Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124603#comment-13124603 ] Jeff Bean commented on MAPREDUCE-2905: -- Verified that this fix works. Can someone please look at it? CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job) --- Key: MAPREDUCE-2905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/fair-share Affects Versions: 0.20.2 Reporter: Jeff Bean Attachments: MR-2905.patch We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster. Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune. It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3158) Fix trunk build failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124601#comment-13124601 ] Hudson commented on MAPREDUCE-3158: --- Integrated in Hadoop-Common-trunk-Commit #1051 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1051/]) MAPREDUCE-3158. Fix test failures in MRv1 due to default framework being set to yarn. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1181310 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ReduceTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/HadoopTestCase.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCollect.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestComparators.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJavaSerialization.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMapOutputType.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMapRed.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestClientProtocolProviderImpls.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestMapCollection.java Fix trunk build failures Key: MAPREDUCE-3158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 Attachments: MR-3158.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3158) Fix trunk build failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124602#comment-13124602 ] Hudson commented on MAPREDUCE-3158: --- Integrated in Hadoop-Hdfs-trunk-Commit #1129 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1129/]) MAPREDUCE-3158. Fix test failures in MRv1 due to default framework being set to yarn. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1181310 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ReduceTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/HadoopTestCase.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCollect.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestComparators.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJavaSerialization.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMapOutputType.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMapRed.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestClientProtocolProviderImpls.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestMapCollection.java Fix trunk build failures Key: MAPREDUCE-3158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 Attachments: MR-3158.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124622#comment-13124622 ] Arun C Murthy commented on MAPREDUCE-2789: -- Thanks Eric. Some minor nits: # Need a null check for 'currentAttempt' in RMAppImpl. # In RMAppAttemptImpl.getApplicationResourceUsageReport: ## You need to acquire the readLock. ## Reserved containers loop needs a +=, not = # In ApplicationResourceUsageReport, please use Resource rather than int for all parameters/return-values. We don't want to tie it to just memory. # JobStatus ## Minor nit - please use 'Reserved' rather than 'Rsvd' for apis, we try to avoid these in api names (maybe use expansion for variable names too?) ## I don't get why we need (get,set)ApplicationMasterInfo - we already have (get,set)TrackingUrl? [MR:279] Update the scheduling info on CLI -- Key: MAPREDUCE-2789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Eric Payne Fix For: 0.23.0, 0.24.0 Attachments: MAPREDUCE-2789.v1.txt, MAPREDUCE-2789.v2.txt The scheduling information such as number of containers running, memory usage and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3158) Fix trunk build failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124621#comment-13124621 ] Hudson commented on MAPREDUCE-3158: --- Integrated in Hadoop-Mapreduce-0.23-Build #43 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/43/]) Merge -c 1181310 from trunk to branch-0.23 to fix MAPREDUCE-3158. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1181311 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ReduceTask.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/HadoopTestCase.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCollect.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestComparators.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJavaSerialization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMapOutputType.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMapRed.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestClientProtocolProviderImpls.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestMapCollection.java Fix trunk build failures Key: MAPREDUCE-3158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 Attachments: MR-3158.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3158) Fix trunk build failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124623#comment-13124623 ] Hudson commented on MAPREDUCE-3158: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1071 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1071/]) MAPREDUCE-3158. Fix test failures in MRv1 due to default framework being set to yarn. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1181310 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ReduceTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/HadoopTestCase.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCollect.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestComparators.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestFileOutputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJavaSerialization.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMapOutputType.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestMapRed.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestClientProtocolProviderImpls.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/TestMapCollection.java Fix trunk build failures Key: MAPREDUCE-3158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Fix For: 0.23.0 Attachments: MR-3158.1.patch https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124647#comment-13124647 ] Luke Lu commented on MAPREDUCE-2858: Thanks Arun and Robert for helping refreshing my memory :) Note, the above list is incomplete e.g., miss an important optimization that by pass the filtering if the request user is the same as the AM, which is 90% of the case; and an important deployment recommendation for even better security than the current 0.20-security deployment. I'll post the current design doc soon. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and container-init event types in NM's ApplicationImpl FSM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-3162: --- Attachment: mr-3162.txt This patch depends on MR-3161 Separate application-init and container-init event types in NM's ApplicationImpl FSM Key: MAPREDUCE-3162 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: mr-3162.txt Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3159) DefaultContainerExecutor removes appcache dir on every localization
[ https://issues.apache.org/jira/browse/MAPREDUCE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-3159: --- Attachment: mr-3159.txt This patch goes on top of MAPREDUCE-3162. With this patch I'm getting a lot farther along in running terasort on 20 nodes with DCE. DefaultContainerExecutor removes appcache dir on every localization --- Key: MAPREDUCE-3159 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3159 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: mr-3159.txt The DefaultContainerExecutor currently has code that removes the application dir from appcache/ in the local directories on every task localization. This causes any concurrent executing tasks from the same job to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3163) JobClient spews errors when killing MR2 job
JobClient spews errors when killing MR2 job --- Key: MAPREDUCE-3163 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3163 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Mahadev konar Priority: Critical When I used the hadoop job command line to kill a running MR2 job, I got a bunch of error spew on the console, despite the kill actually taking effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3163) JobClient spews errors when killing MR2 job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-3163: --- Attachment: log.txt Here's the console log JobClient spews errors when killing MR2 job --- Key: MAPREDUCE-3163 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3163 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Mahadev konar Priority: Critical Attachments: log.txt When I used the hadoop job command line to kill a running MR2 job, I got a bunch of error spew on the console, despite the kill actually taking effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3159) DefaultContainerExecutor removes appcache dir on every localization
[ https://issues.apache.org/jira/browse/MAPREDUCE-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-3159: --- Attachment: mr-3159.txt oops, missed a file. DefaultContainerExecutor removes appcache dir on every localization --- Key: MAPREDUCE-3159 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3159 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: mr-3159.txt, mr-3159.txt The DefaultContainerExecutor currently has code that removes the application dir from appcache/ in the local directories on every task localization. This causes any concurrent executing tasks from the same job to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and container-init event types in NM's ApplicationImpl FSM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124678#comment-13124678 ] Todd Lipcon commented on MAPREDUCE-3162: there's a bit of a bug in this right now: startContainer can be called concurrently from several threads, apparently. So one thread can insert the ApplicationImpl in the concurrent applications map, then another thread can send the INIT_CONTAINER event before the INIT_APPLICATION event has been sent. Thinking about a good fix. Separate application-init and container-init event types in NM's ApplicationImpl FSM Key: MAPREDUCE-3162 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: mr-3162.txt Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124680#comment-13124680 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3161: Straightforward patch. Looks good. +1. Thanks for resisting a developer's temptation of fixing code-bugs along with typos ;) Regarding the docs for the transitions (and also for events), I think decorating them with annotations is really useful. For e.g, (start state, end state) for transitions and (source, destination) pair for events. Will file a separate ticket, I am fine passing that for this patch. Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3161: --- Fix Version/s: 0.23.0 Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3161: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this to trunk and branch-0.23. Thanks Todd! Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124681#comment-13124681 ] Todd Lipcon commented on MAPREDUCE-3161: Thanks Vinod. I'll commit this soon. I was also thinking about an annotation-based API - does one already exist? Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2858) MRv2 WebApp Security
[ https://issues.apache.org/jira/browse/MAPREDUCE-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124684#comment-13124684 ] Arun C Murthy commented on MAPREDUCE-2858: -- Luke, we are running against time. Can I ask Robert to get started on this one to ensure we make it soon? Thanks. MRv2 WebApp Security Key: MAPREDUCE-2858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2858 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: applicationmaster, mrv2, security Affects Versions: 0.23.0 Reporter: Luke Lu Assignee: Luke Lu Priority: Blocker Fix For: 0.23.0 In MRv2, while the system servers (ResourceManager (RM), NodeManager (NM) and NameNode (NN)) run as trusted system users, the application masters (AM) run as users who submit the application. While this offers great flexibility to run multiple version of mapreduce frameworks (including their UI) on the same Hadoop cluster, it has significant implication for the security of webapps (Please do not discuss company specific vulnerabilities here). Requirements: # Secure authentication for AM (for app/job level ACLs). # Webapp security should be optional via site configuration. # Support existing pluggable single sign on mechanisms. # Should not require per app/user configuration for deployment. # Should not require special site-wide DNS configuration for deployment. This the top jira for webapp security. A design doc/notes of threat-modeling and counter measures will be posted on the wiki. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124683#comment-13124683 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3161: Asynchronous events and race conditions :) bq. I was also thinking about an annotation-based API - does one already exist? Nope, not yet. Let me create a ticket. Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124686#comment-13124686 ] Hudson commented on MAPREDUCE-3161: --- Integrated in Hadoop-Common-trunk-Commit #1053 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1053/]) MAPREDUCE-3161. Improved some javadocs and fixed some typos in YARN. Contributed by Todd Lipcon. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1181622 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/ExponentiallySmoothedTaskRuntimeEstimator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/ContainerLocalizationRequestEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/LocalizationEvent.java Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124687#comment-13124687 ] Hudson commented on MAPREDUCE-3161: --- Integrated in Hadoop-Hdfs-trunk-Commit #1131 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1131/]) MAPREDUCE-3161. Improved some javadocs and fixed some typos in YARN. Contributed by Todd Lipcon. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1181622 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/ExponentiallySmoothedTaskRuntimeEstimator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/ContainerLocalizationRequestEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/LocalizationEvent.java Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and container-init event types in NM's ApplicationImpl FSM
[
https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124688#comment-13124688
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3162:
Yes, that was the reason why Chris/I originally made INIT_APPLICATION
idempotent and handled it like this. This is a manifestation of the fact that
we don't have a separate {{startApplication()}} API along with
{{startContainer()}}.
I can understand the clarity of separate events, but not sure if there is any
way out. Will let you reach to the conclusion ;)
Separate application-init and container-init event types in NM's
ApplicationImpl FSM
Key: MAPREDUCE-3162
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
Attachments: mr-3162.txt
Currently, the ApplicationImpl receives an INIT_APPLICATION event on every
container initialization. Only on the first one does it really mean to init
the application, whereas all subsequent events are for specific containers.
This JIRA is to separate the events into INIT_APPLICATION, sent once and only
once per application, and INIT_CONTAINER, which is sent for every container.
The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3164) Decorate event transitions and the event-types with their behaviour
Decorate event transitions and the event-types with their behaviour --- Key: MAPREDUCE-3164 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3164 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Helps to annotate the transitions with (start-state, end-state) pair and the events with (source, destination) pair. Not just readability, we may also use them to generate the event diagrams across components. Not a blocker for 0.23, but let's see. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124692#comment-13124692 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3161: Filed MAPREDUCE-3164 for decorating transitions and events. Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124693#comment-13124693 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2666: Thanks Jonathan. +1 for doing it via JobHistory instead of adding a new API to NodeManager. MR-279: Need to retrieve shuffle port number on ApplicationMaster restart - Key: MAPREDUCE-2666 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.0 MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on. In the case of an ApplicationMaster crash where it needs to be restarted that information is lost. We either need to re-query it from each of the NodeManagers or to persist it to the JobHistory logs and retrieve it again. The job history logs is probably the simpler solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3157) Rumen TraceBuilder is skipping analyzing 0.20 history files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124694#comment-13124694 ] Amar Kamat commented on MAPREDUCE-3157: --- Ravi, Instead of parsing the current job-history files using JobHistory.CONF_FILENAME_REGEX or JobHistory.JOBHISTORY_FILENAME_REGEX, should we use the JobHistory's filename parsing APIs like getJobIDFromHistoryFilePath()? Rumen TraceBuilder is skipping analyzing 0.20 history files --- Key: MAPREDUCE-3157 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3157 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: MR3157.patch Rumen TraceBuilder is assuming the Pre21 history file name format to be JTIdentifier_jobId_something. But it can be jobId_something also as it is now in latest 0.20.x version. This also needs to be understood by TraceBuilder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3157) Rumen TraceBuilder is skipping analyzing 0.20 history files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124695#comment-13124695 ] Amar Kamat commented on MAPREDUCE-3157: --- Other than my previous comment, the patch looks good to me. Rumen TraceBuilder is skipping analyzing 0.20 history files --- Key: MAPREDUCE-3157 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3157 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 0.23.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.23.0 Attachments: MR3157.patch Rumen TraceBuilder is assuming the Pre21 history file name format to be JTIdentifier_jobId_something. But it can be jobId_something also as it is now in latest 0.20.x version. This also needs to be understood by TraceBuilder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and container-init event types in NM's ApplicationImpl FSM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124696#comment-13124696 ] Chris Douglas commented on MAPREDUCE-3162: -- bq. startContainer can be called concurrently from several threads, apparently. So one thread can insert the ApplicationImpl in the concurrent applications map, then another thread can send the INIT_CONTAINER event before the INIT_APPLICATION event has been sent. That was the intent. It should also be legal for the NM to clean up application state, given a richer set of semantics for the intermediate output specifying retention. Then one would have to re-init the application to start containers there. More of the init used to be per-application, before changing the localization to be per-container. It's not doing much now, but it's cheap. Separate application-init and container-init event types in NM's ApplicationImpl FSM Key: MAPREDUCE-3162 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: mr-3162.txt Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3161) Improve javadoc and fix some typos in MR2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124697#comment-13124697 ] Hudson commented on MAPREDUCE-3161: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1073 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1073/]) MAPREDUCE-3161. Improved some javadocs and fixed some typos in YARN. Contributed by Todd Lipcon. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1181622 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/ExponentiallySmoothedTaskRuntimeEstimator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalResourcesTrackerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/ContainerLocalizationRequestEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/event/LocalizationEvent.java Improve javadoc and fix some typos in MR2 code -- Key: MAPREDUCE-3161 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3161 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 0.23.0 Attachments: mr-3161.txt Just some simple cleanup, documentation, typos in variable names, etc. The only code change is to refactor ResourceLocalizationService so each event type is handled in its own method instead of a giant switch statement (just using eclipse's Extract Method - no semantic change) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and container-init event types in NM's ApplicationImpl FSM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13124699#comment-13124699 ] Todd Lipcon commented on MAPREDUCE-3162: Thanks for the comments. I fixed this locally by handling INIT_CONTAINER events during NEW state the same as during INITING (ie by queueing them up). Do you think that's worse? The whole DuplicateAppInitTransition thing was very confusing to me as I first looked at the code. Separate application-init and container-init event types in NM's ApplicationImpl FSM Key: MAPREDUCE-3162 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: mr-3162.txt Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
