[jira] [Commented] (MAPREDUCE-3545) Remove Avro RPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185427#comment-13185427 ] Hudson commented on MAPREDUCE-3545: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1556 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1556/]) Remove the empty avro directories for MAPREDUCE-3545. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230886 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/avro * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/avro * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/avro * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/avro > Remove Avro RPC > --- > > Key: MAPREDUCE-3545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3545 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.1, 0.24.0 > > Attachments: MR-3545.txt > > > Please see the discussion in HDFS-2660 for more details. I have created a > branch HADOOP-6659 to save the Avro work, if in the future some one wants to > use the work that existed to add support for Avro RPC. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3545) Remove Avro RPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185421#comment-13185421 ] Eli Collins commented on MAPREDUCE-3545: Yes, we must have missed that. Thanks Nicholas. > Remove Avro RPC > --- > > Key: MAPREDUCE-3545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3545 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.1, 0.24.0 > > Attachments: MR-3545.txt > > > Please see the discussion in HDFS-2660 for more details. I have created a > branch HADOOP-6659 to save the Avro work, if in the future some one wants to > use the work that existed to add support for Avro RPC. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3545) Remove Avro RPC
[
https://issues.apache.org/jira/browse/MAPREDUCE-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185420#comment-13185420
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-3545:
---
BTW, there is a non-empty avro src directory. Should it be removed?
{noformat}
$ls
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/avro
Events.avpr
{noformat}
> Remove Avro RPC
> ---
>
> Key: MAPREDUCE-3545
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3545
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3545.txt
>
>
> Please see the discussion in HDFS-2660 for more details. I have created a
> branch HADOOP-6659 to save the Avro work, if in the future some one wants to
> use the work that existed to add support for Avro RPC.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3545) Remove Avro RPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185419#comment-13185419 ] Hudson commented on MAPREDUCE-3545: --- Integrated in Hadoop-Common-trunk-Commit #1538 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1538/]) Remove the empty avro directories for MAPREDUCE-3545. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230886 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/avro * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/avro * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/avro * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/avro > Remove Avro RPC > --- > > Key: MAPREDUCE-3545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3545 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.1, 0.24.0 > > Attachments: MR-3545.txt > > > Please see the discussion in HDFS-2660 for more details. I have created a > branch HADOOP-6659 to save the Avro work, if in the future some one wants to > use the work that existed to add support for Avro RPC. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3545) Remove Avro RPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185418#comment-13185418 ] Hudson commented on MAPREDUCE-3545: --- Integrated in Hadoop-Hdfs-trunk-Commit #1611 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1611/]) Remove the empty avro directories for MAPREDUCE-3545. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230886 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/avro * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/avro * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/avro * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/avro > Remove Avro RPC > --- > > Key: MAPREDUCE-3545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3545 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.1, 0.24.0 > > Attachments: MR-3545.txt > > > Please see the discussion in HDFS-2660 for more details. I have created a > branch HADOOP-6659 to save the Avro work, if in the future some one wants to > use the work that existed to add support for Avro RPC. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3545) Remove Avro RPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185414#comment-13185414 ] Tsz Wo (Nicholas), SZE commented on MAPREDUCE-3545: --- The empty avro directories should also be removed. Let me do it now. > Remove Avro RPC > --- > > Key: MAPREDUCE-3545 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3545 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.1 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.1, 0.24.0 > > Attachments: MR-3545.txt > > > Please see the discussion in HDFS-2660 for more details. I have created a > branch HADOOP-6659 to save the Avro work, if in the future some one wants to > use the work that existed to add support for Avro RPC. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185394#comment-13185394
]
Hadoop QA commented on MAPREDUCE-3596:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12510446/MAPREDUCE-3596-20120112.1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1609//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1609//console
This message is automatically generated.
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.1.txt,
> MAPREDUCE-3596-20120112.txt, logs.tar.bz2, logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-177) Hadoop performance degrades significantly as more and more jobs complete
[ https://issues.apache.org/jira/browse/MAPREDUCE-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185376#comment-13185376 ] Evan Pollan commented on MAPREDUCE-177: --- This defect affects me after just 10-15 iterations of a daily job that has on the order to 10K mappers and a thousand or so reducers. This is cropping up using 0.20.2 (from CDH2U3). This seems like a pretty serious problem affecting the longevity of the job tracker. Is there a reason a fix hasn't been committed and released? > Hadoop performance degrades significantly as more and more jobs complete > > > Key: MAPREDUCE-177 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-177 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Runping Qi >Assignee: Ioannis Koltsidas >Priority: Critical > Attachments: HADOOP-4766-v1.patch, HADOOP-4766-v2.10.patch, > HADOOP-4766-v2.4.patch, HADOOP-4766-v2.6.patch, HADOOP-4766-v2.7-0.18.patch, > HADOOP-4766-v2.7-0.19.patch, HADOOP-4766-v2.7.patch, > HADOOP-4766-v2.8-0.18.patch, HADOOP-4766-v2.8-0.19.patch, > HADOOP-4766-v2.8.patch, HADOOP-4766-v3.4-0.19.patch, map_scheduling_rate.txt > > > When I ran the gridmix 2 benchmark load on a fresh cluster of 500 nodes with > hadoop trunk, > the gridmix load, consisting of 202 map/reduce jobs of various sizes, > completed in 32 minutes. > Then I ran the same set of the jobs on the same cluster, yhey completed in 43 > minutes. > When I ran them the third times, it took (almost) forever --- the job tracker > became non-responsive. > The job tracker's heap size was set to 2GB. > The cluster is configured to keep up to 500 jobs in memory. > The job tracker kept one cpu busy all the time. Look like it was due to GC. > I believe the release 0.18/0.19 have the similar behavior. > I believe 0.18 and 0.18 also have the similar behavior. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3596:
---
Attachment: MAPREDUCE-3596-20120112.1.txt
Updated patch. Making scheduler inform the Node about containers that it
doesn't know about.
Fortunately I could change my unit tests a bit to reproduce the corner case
also. The test hangs without the fix and passes with.
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.1.txt,
> MAPREDUCE-3596-20120112.txt, logs.tar.bz2, logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3596:
---
Hadoop Flags: (was: Reviewed)
Status: Patch Available (was: Open)
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.1.txt,
> MAPREDUCE-3596-20120112.txt, logs.tar.bz2, logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185356#comment-13185356
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3596:
Thanks Robert.
And a very good catch Sid. Doing it on NM is more complicated with the
heartbeat thread different from the AMNM RPC. I am tending to do it on the RM
itself inside the scheduler.
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.txt, logs.tar.bz2,
> logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[
https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3641:
-
Attachment: MAPREDUCE-3641.patch
Update patch with some bug-fixes and some extra debug info (not for commit).
I'm still hunting a NPE down:
{noformat}
12/01/12 23:47:38 ERROR resourcemanager.ResourceManager: Error in handling
event type NODE_UPDATE to the schedulerjava.lang.NullPointerExceptionat
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp.unreserve(SchedulerApp.java:392)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.unreserve(LeafQueue.java:1217)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1158)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignRackLocalContainers(LeafQueue.java:1027)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:989)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignReservedContainer(LeafQueue.java:821)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:721)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:568)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:615)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:295)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I'm also seeing negative container counts for the queue etc.
Need to dig some more.
> CapacityScheduler should be more conservative assigning off-switch requests
> ---
>
> Key: MAPREDUCE-3641
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, scheduler
>Affects Versions: 0.23.0
>Reporter: Arun C Murthy
>Assignee: Arun C Murthy
>Priority: Blocker
> Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch
>
>
> In hadoop-1, the CS is very conservative handing out off-switch assignments,
> we need to do the same in YARN.
> We noticed performance regressions due to this, particularly for reduces.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-3596:
--
Status: Open (was: Patch Available)
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.txt, logs.tar.bz2,
> logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185313#comment-13185313
]
Siddharth Seth commented on MAPREDUCE-3596:
---
The NPE could still happen if the startContainer request to the NM is delayed.
The patch fixes the case where a newly launched container is reported in a NM
heartbeat and the RM is still aware that this container needs to be cleaned up.
If the newly launched container (because of a delayed startContainer) is sent
to the RM in a subsequent heartbeat (after the RM has told it to clean up the
container and cleaned up it's own list of containersToCleanup) - will end up in
the same NPE, and the container running to completion.
One possible option would be to have the NM keep track of containers it needs
to clean up - if it isn't aware of the container yet.
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.txt, logs.tar.bz2,
> logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3662) Command line ask: NM info where containers are launched
[
https://issues.apache.org/jira/browse/MAPREDUCE-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash resolved MAPREDUCE-3662.
-
Resolution: Duplicate
Marking as Duplicate. I'd forgot I'd opened MAPREDUCE-3406
> Command line ask: NM info where containers are launched
> ---
>
> Key: MAPREDUCE-3662
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3662
> Project: Hadoop Map/Reduce
> Issue Type: Wish
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>
> Courtesy [~rramya]
> {quote}
> we had requested for the NM information where the containers are scheduled to
> be made available in job
> -list-attempt-ids. This will be helpful in automation, debugging and avoid
> grepping through the AM logs.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3406) Add node information to bin/mapred job -list-attempt-ids and other improvements
[ https://issues.apache.org/jira/browse/MAPREDUCE-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash resolved MAPREDUCE-3406. - Resolution: Duplicate Marking as Duplicate. I'd forgot I'd opened MAPREDUCE-3406 > Add node information to bin/mapred job -list-attempt-ids and other > improvements > --- > > Key: MAPREDUCE-3406 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3406 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.23.1 > > > From [~rramya] > Providing the NM information where the containers are scheduled in bin/mapred > job -list-attempt-ids will be helpful in automation, debugging and to avoid > grepping through the AM logs. > From my own observation, the list-attempt-ids should list the attempt ids and > not require the arguments. The arguments if given, can be used to filter the > results. From the usage: > bq. [-list-attempt-ids ]. Valid values for > are MAP REDUCE JOB_SETUP JOB_CLEANUP TASK_CLEANUP. Valid values > for are running, completed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-3406) Add node information to bin/mapred job -list-attempt-ids and other improvements
[ https://issues.apache.org/jira/browse/MAPREDUCE-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash reopened MAPREDUCE-3406: - > Add node information to bin/mapred job -list-attempt-ids and other > improvements > --- > > Key: MAPREDUCE-3406 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3406 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.23.1 > > > From [~rramya] > Providing the NM information where the containers are scheduled in bin/mapred > job -list-attempt-ids will be helpful in automation, debugging and to avoid > grepping through the AM logs. > From my own observation, the list-attempt-ids should list the attempt ids and > not require the arguments. The arguments if given, can be used to filter the > results. From the usage: > bq. [-list-attempt-ids ]. Valid values for > are MAP REDUCE JOB_SETUP JOB_CLEANUP TASK_CLEANUP. Valid values > for are running, completed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3404) Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false
[ https://issues.apache.org/jira/browse/MAPREDUCE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated MAPREDUCE-3404: -- Fix Version/s: 0.24.0 > Speculative Execution: speculative map tasks launched even if > -Dmapreduce.map.speculative=false > --- > > Key: MAPREDUCE-3404 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3404 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, mrv2 >Affects Versions: 0.23.0 > Environment: Hadoop version is: Hadoop 0.23.0.1110031628 > 10 node test cluster >Reporter: patrick white >Assignee: Eric Payne >Priority: Critical > Fix For: 0.23.0, 0.23.1, 0.24.0 > > Attachments: MAPREDUCE-3404.1.txt, MAPREDUCE-3404.2.txt > > > When forcing a mapper to take significantly longer than other map tasks, > speculative map tasks are > launched even if the mapreduce.job.maps.speculative.execution parameter is > set to 'false'. > Testcase: ran default WordCount job with spec execution set to false for both > map and reduce but still saw a fifth mapper > task launch, ran job as follows: > hadoop --config jar /tmp/testphw/wordcount.jar WordCount > -Dmapreduce.job.maps.speculative.execution=false > -Dmapreduce.job.reduces.speculative.execution=false > /tmp/test_file_of_words* /tmp/file_of_words.out > Input data was 4 text files >hdfs blocksize, with same word pattern plus one > diff text line in each file, fourth > file was 4 times as large as others: > hadoop --config fs -ls /tmp > Found 5 items > drwxr-xr-x - user hdfs 0 2011-10-20 16:17 /tmp/file_of_words.out > -rw-r--r-- 3 user hdfs 62800021 2011-10-20 14:45 /tmp/test_file_of_words1 > -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words2 > -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words3 > -rw-r--r-- 3 user hdfs 271708312 2011-10-20 15:50 /tmp/test_file_of_words4 > Job launched 5 mappers despite spec exec set to false, output snippet: > org.apache.hadoop.mapreduce.JobCounter > NUM_FAILED_MAPS=1 > TOTAL_LAUNCHED_MAPS=5 > TOTAL_LAUNCHED_REDUCES=1 > RACK_LOCAL_MAPS=5 > SLOTS_MILLIS_MAPS=273540 > SLOTS_MILLIS_REDUCES=212876 > Reran same case as above only set both spec exec params to 'true', same > results only this time the fifth task being > launched is expected since spec exec = true. > job run: > hadoop --config jar /tmp/testphw/wordcount.jar WordCount > -Dmapreduce.job.maps.speculative.execution=true > -Dmapreduce.job.reduces.speculative.execution=true > /tmp/test_file_of_words* /tmp/file_of_words.out > output snippet: > org.apache.hadoop.mapreduce.JobCounter > NUM_FAILED_MAPS=1 > TOTAL_LAUNCHED_MAPS=5 > TOTAL_LAUNCHED_REDUCES=1 > RACK_LOCAL_MAPS=5 > SLOTS_MILLIS_MAPS=279653 > SLOTS_MILLIS_REDUCES=211474 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185290#comment-13185290
]
Hadoop QA commented on MAPREDUCE-3656:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12510427/MR3656.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1608//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1608//console
This message is automatically generated.
> Sort job on 350 scale is consistently failing with latest MRV2 code
>
>
> Key: MAPREDUCE-3656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2, resourcemanager
>Affects Versions: 0.23.1
>Reporter: Karam Singh
>Assignee: Siddharth Seth
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3656.txt, MR3656.txt
>
>
> With the code checked out on last two days.
> Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
> failing for around last 6 runs
> When around 50% of maps are completed, suddenly job jumps to failed state.
> On looking at NM log, found RM sent Stop Container Request to NM for AM
> container.
> But at INFO level from RM log not able find why RM is killing AM when job is
> not killed manually.
> One thing found common on failed AM logs is -:
> org.apache.hadoop.yarn.state.InvalidStateTransitonException
> With with different.
> For e.g. One log says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_UPDATE at ASSIGNED
> {code}
> Whereas other logs says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_COUNTER_UPDATE at ERROR
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185268#comment-13185268 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3360: Looking at the patch for final review/commit. Completely missed this ticket and all the conversation that happened, will take a little bit of time. > Provide information about lost nodes in the UI. > --- > > Key: MAPREDUCE-3360 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 > Environment: NA >Reporter: Bhallamudi Venkata Siva Kamesh >Assignee: Bhallamudi Venkata Siva Kamesh >Priority: Critical > Attachments: LostNodes.png, MAPREDUCE-3360-1.patch, > MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch, > MAPREDUCE-3360-5.patch, MAPREDUCE-3360-6.patch, MAPREDUCE-3360.patch, > lostNodes.png > > > Currently there is no information provided about *lost nodes*. Provide > information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185256#comment-13185256
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3656:
+1. This looks good now.
> Sort job on 350 scale is consistently failing with latest MRV2 code
>
>
> Key: MAPREDUCE-3656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2, resourcemanager
>Affects Versions: 0.23.1
>Reporter: Karam Singh
>Assignee: Siddharth Seth
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3656.txt, MR3656.txt
>
>
> With the code checked out on last two days.
> Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
> failing for around last 6 runs
> When around 50% of maps are completed, suddenly job jumps to failed state.
> On looking at NM log, found RM sent Stop Container Request to NM for AM
> container.
> But at INFO level from RM log not able find why RM is killing AM when job is
> not killed manually.
> One thing found common on failed AM logs is -:
> org.apache.hadoop.yarn.state.InvalidStateTransitonException
> With with different.
> For e.g. One log says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_UPDATE at ASSIGNED
> {code}
> Whereas other logs says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_COUNTER_UPDATE at ERROR
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-3656:
--
Status: Patch Available (was: Open)
> Sort job on 350 scale is consistently failing with latest MRV2 code
>
>
> Key: MAPREDUCE-3656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2, resourcemanager
>Affects Versions: 0.23.1
>Reporter: Karam Singh
>Assignee: Siddharth Seth
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3656.txt, MR3656.txt
>
>
> With the code checked out on last two days.
> Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
> failing for around last 6 runs
> When around 50% of maps are completed, suddenly job jumps to failed state.
> On looking at NM log, found RM sent Stop Container Request to NM for AM
> container.
> But at INFO level from RM log not able find why RM is killing AM when job is
> not killed manually.
> One thing found common on failed AM logs is -:
> org.apache.hadoop.yarn.state.InvalidStateTransitonException
> With with different.
> For e.g. One log says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_UPDATE at ASSIGNED
> {code}
> Whereas other logs says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_COUNTER_UPDATE at ERROR
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185245#comment-13185245
]
Siddharth Seth commented on MAPREDUCE-3656:
---
Updating the patch with the review comments.
bq. For handling the case where JVM is unregistered before it gets a task, we
should remove it from launchedJVMs during unregister. Once we do this, we
should think about synchronization issues carefully.
Good catch. Uploading a patch which removes the jvm from the launchedJVMs set
in the unregister call, prior to removing from jvmIDToActiveAttemptMap. The
ordering of events between unregister and getTask should take care of
synchronization issues.
bq. We went through a couple of iterations on this part of the code, so let us
make sure things are fine by running the AMScalability benchmark (100K maps)
once.
Have already run a sort benchmark with the previous patch and the patch from
MAPREDUCE-3596, which passed. Can run AMScalability as well - but this issue
has never been seen with AMScalability (shows up primarily when shuffle starts
and the startContainer calls slow down).
Another change which can be made is to have TaskAttemptListener /
TaskHeartbeatHandler throw Exceptions for calls from unregistered tasks.
Currently the AM relies on the NM stopContainer to kill these tasks. Opening a
separate jira for this. Also one for the NM startContainer calls slowing down.
> Sort job on 350 scale is consistently failing with latest MRV2 code
>
>
> Key: MAPREDUCE-3656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2, resourcemanager
>Affects Versions: 0.23.1
>Reporter: Karam Singh
>Assignee: Siddharth Seth
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3656.txt, MR3656.txt
>
>
> With the code checked out on last two days.
> Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
> failing for around last 6 runs
> When around 50% of maps are completed, suddenly job jumps to failed state.
> On looking at NM log, found RM sent Stop Container Request to NM for AM
> container.
> But at INFO level from RM log not able find why RM is killing AM when job is
> not killed manually.
> One thing found common on failed AM logs is -:
> org.apache.hadoop.yarn.state.InvalidStateTransitonException
> With with different.
> For e.g. One log says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_UPDATE at ASSIGNED
> {code}
> Whereas other logs says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_COUNTER_UPDATE at ERROR
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-3656:
--
Attachment: MR3656.txt
> Sort job on 350 scale is consistently failing with latest MRV2 code
>
>
> Key: MAPREDUCE-3656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2, resourcemanager
>Affects Versions: 0.23.1
>Reporter: Karam Singh
>Assignee: Siddharth Seth
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3656.txt, MR3656.txt
>
>
> With the code checked out on last two days.
> Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
> failing for around last 6 runs
> When around 50% of maps are completed, suddenly job jumps to failed state.
> On looking at NM log, found RM sent Stop Container Request to NM for AM
> container.
> But at INFO level from RM log not able find why RM is killing AM when job is
> not killed manually.
> One thing found common on failed AM logs is -:
> org.apache.hadoop.yarn.state.InvalidStateTransitonException
> With with different.
> For e.g. One log says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_UPDATE at ASSIGNED
> {code}
> Whereas other logs says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_COUNTER_UPDATE at ERROR
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3650) testGetTokensForHftpFS() fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp reassigned MAPREDUCE-3650: -- Assignee: Daryn Sharp > testGetTokensForHftpFS() fails > -- > > Key: MAPREDUCE-3650 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3650 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Daryn Sharp >Priority: Blocker > Fix For: 0.23.0 > > > org.apache.hadoop.mapreduce.security.TestTokenCache.testGetTokensForHftpFS > fails. > Looks like it may have been introduced with HADOOP-7808 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3377) Compatibility issue with 0.20.203.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned MAPREDUCE-3377: -- Assignee: Jane Chen > Compatibility issue with 0.20.203. > -- > > Key: MAPREDUCE-3377 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.203.0, 1.0.0 > Environment: 0.20.203.0 standalone JDK 1.6 Linux >Reporter: Jane Chen >Assignee: Jane Chen > Attachments: mapreduce-3377.patch > > > I have an OutputFormat which implements Configurable. I set new config > entries to a job configuration during checkOutputSpec() so that the tasks > will get the config entries through the job configuration. This works fine > in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my > OutputFormat still has the configuration set, but the copy a task gets does > not have the new entries that are set as part of checkOutputSpec(). > I believe that the problem is with JobClient. The job configuration needs to > wait till checkOutputSpec() is returned before being cloned and submitted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3377) Compatibility issue with 0.20.203.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185213#comment-13185213 ] Eli Collins commented on MAPREDUCE-3377: Hi Jane, You change looks good to me. Would you mind adding a test that would fail currently w/o your patch, and would regress if it was removed? Thanks, Eli > Compatibility issue with 0.20.203. > -- > > Key: MAPREDUCE-3377 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.203.0, 1.0.0 > Environment: 0.20.203.0 standalone JDK 1.6 Linux >Reporter: Jane Chen > Attachments: mapreduce-3377.patch > > > I have an OutputFormat which implements Configurable. I set new config > entries to a job configuration during checkOutputSpec() so that the tasks > will get the config entries through the job configuration. This works fine > in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my > OutputFormat still has the configuration set, but the copy a task gets does > not have the new entries that are set as part of checkOutputSpec(). > I believe that the problem is with JobClient. The job configuration needs to > wait till checkOutputSpec() is returned before being cloned and submitted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3666) Broken documentation link for r0.23.0
Broken documentation link for r0.23.0 - Key: MAPREDUCE-3666 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3666 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Reporter: Jason Lowe Priority: Minor Apache Hadoop website has a broken link for the r0.23.0 release (http://hadoop.apache.org/mapreduce/docs/r0.23.0/). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185196#comment-13185196
]
Robert Joseph Evans commented on MAPREDUCE-3596:
I looked at the new patch and I am still +1 (non-binding), just to be official,
because it is a new patch.
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.txt, logs.tar.bz2,
> logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3582) Move successfully passing MR1 tests to MR2 maven tree.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185195#comment-13185195 ] Hadoop QA commented on MAPREDUCE-3582: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510417/MR-3582rev7fixes.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1607//console This message is automatically generated. > Move successfully passing MR1 tests to MR2 maven tree. > -- > > Key: MAPREDUCE-3582 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3582 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3582-part1_rev3.patch, > MAPREDUCE-3582-part1_rev4.patch, MAPREDUCE-3582-part1_rev5.patch, > MAPREDUCE-3582-part1_rev6.patch, MAPREDUCE-3582-part1_rev7.patch, > MAPREDUCE-3582-part2_rev3.patch, MAPREDUCE-3582-part2_rev4.patch, > MAPREDUCE-3582-part2_rev5.patch, MAPREDUCE-3582-part2_rev6.patch, > MAPREDUCE-3582-part2_rev7.patch, MAPREDUCE-3582-whole_rev3.patch, > MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev4.patch, > MAPREDUCE-3582-whole_rev5.patch, MAPREDUCE-3582-whole_rev6.patch, > MAPREDUCE-3582-whole_rev7.patch, MAPREDUCE-3582.patch, > MAPREDUCE-3582_rev2.patch, MR-3582rev7fixes.patch, mv_script_MR-3582.sh, > mv_script_MR-3582_rev2.sh, mv_script_MR-3582_rev3.sh, > mv_script_MR-3582_rev4.sh, mv_script_MR-3582_rev5.sh, > mv_script_MR-3582_rev6.sh, mv_script_MR-3582_rev7.sh > > > This ticket will track moving mr1 tests that are passing successfully to mr2 > maven tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185194#comment-13185194
]
Hadoop QA commented on MAPREDUCE-3596:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12510415/MAPREDUCE-3596-20120112.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1606//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1606//console
This message is automatically generated.
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.txt, logs.tar.bz2,
> logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185191#comment-13185191 ] Hudson commented on MAPREDUCE-3664: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1555 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1555/]) MAPREDUCE-3664. Federation Documentation has incorrect configuration example. Contributed by Brandon Li. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230708 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3582) Move successfully passing MR1 tests to MR2 maven tree.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3582: -- Attachment: MR-3582rev7fixes.patch Ahmed, from build perspective it looks good. Just a couple of minor tweaks (attached rev7fixes patch) that set the scope of hadoop deps to provided. > Move successfully passing MR1 tests to MR2 maven tree. > -- > > Key: MAPREDUCE-3582 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3582 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3582-part1_rev3.patch, > MAPREDUCE-3582-part1_rev4.patch, MAPREDUCE-3582-part1_rev5.patch, > MAPREDUCE-3582-part1_rev6.patch, MAPREDUCE-3582-part1_rev7.patch, > MAPREDUCE-3582-part2_rev3.patch, MAPREDUCE-3582-part2_rev4.patch, > MAPREDUCE-3582-part2_rev5.patch, MAPREDUCE-3582-part2_rev6.patch, > MAPREDUCE-3582-part2_rev7.patch, MAPREDUCE-3582-whole_rev3.patch, > MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev4.patch, > MAPREDUCE-3582-whole_rev5.patch, MAPREDUCE-3582-whole_rev6.patch, > MAPREDUCE-3582-whole_rev7.patch, MAPREDUCE-3582.patch, > MAPREDUCE-3582_rev2.patch, MR-3582rev7fixes.patch, mv_script_MR-3582.sh, > mv_script_MR-3582_rev2.sh, mv_script_MR-3582_rev3.sh, > mv_script_MR-3582_rev4.sh, mv_script_MR-3582_rev5.sh, > mv_script_MR-3582_rev6.sh, mv_script_MR-3582_rev7.sh > > > This ticket will track moving mr1 tests that are passing successfully to mr2 > maven tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3377) Compatibility issue with 0.20.203.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Chen updated MAPREDUCE-3377: - Affects Version/s: 1.0.0 > Compatibility issue with 0.20.203. > -- > > Key: MAPREDUCE-3377 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3377 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.203.0, 1.0.0 > Environment: 0.20.203.0 standalone JDK 1.6 Linux >Reporter: Jane Chen > Attachments: mapreduce-3377.patch > > > I have an OutputFormat which implements Configurable. I set new config > entries to a job configuration during checkOutputSpec() so that the tasks > will get the config entries through the job configuration. This works fine > in 0.20.2, but stopped working starting from 0.20.203. With 0.20.203, my > OutputFormat still has the configuration set, but the copy a task gets does > not have the new entries that are set as part of checkOutputSpec(). > I believe that the problem is with JobClient. The job configuration needs to > wait till checkOutputSpec() is returned before being cloned and submitted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3656:
---
Status: Open (was: Patch Available)
Looks good overall. A couple of minor comments:
- The comment saying "Timing can cause this to happen [..]" needs to be
updated/removed
- For handling the case where JVM is unregistered before it gets a task, we
should remove it from {{launchedJVMs}} during unregister. Once we do this, we
should think about synchronization issues carefully.
- In getTask(), why do we need both the checks
{{jvmIDToActiveAttemptMap.containsKey(wJvmID)}} and
{{jvmIDToActiveAttemptMap.get(wJvmID) == null}}
- We went through a couple of iterations on this part of the code, so let us
make sure things are fine by running the AMScalability benchmark (100K maps)
once.
> Sort job on 350 scale is consistently failing with latest MRV2 code
>
>
> Key: MAPREDUCE-3656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2, resourcemanager
>Affects Versions: 0.23.1
>Reporter: Karam Singh
>Assignee: Siddharth Seth
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR3656.txt
>
>
> With the code checked out on last two days.
> Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
> failing for around last 6 runs
> When around 50% of maps are completed, suddenly job jumps to failed state.
> On looking at NM log, found RM sent Stop Container Request to NM for AM
> container.
> But at INFO level from RM log not able find why RM is killing AM when job is
> not killed manually.
> One thing found common on failed AM logs is -:
> org.apache.hadoop.yarn.state.InvalidStateTransitonException
> With with different.
> For e.g. One log says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_UPDATE at ASSIGNED
> {code}
> Whereas other logs says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_COUNTER_UPDATE at ERROR
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185170#comment-13185170 ] Hudson commented on MAPREDUCE-3664: --- Integrated in Hadoop-Common-trunk-Commit #1537 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1537/]) MAPREDUCE-3664. Federation Documentation has incorrect configuration example. Contributed by Brandon Li. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230708 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185169#comment-13185169 ] Hudson commented on MAPREDUCE-3664: --- Integrated in Hadoop-Hdfs-trunk-Commit #1610 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1610/]) MAPREDUCE-3664. Federation Documentation has incorrect configuration example. Contributed by Brandon Li. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230708 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3596:
---
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.txt, logs.tar.bz2,
> logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3665) Hadoop ignores old-style config options for enabling compressed output
Hadoop ignores old-style config options for enabling compressed output -- Key: MAPREDUCE-3665 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3665 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and mapred.output.compression.codec -- settings that work on 0.20 but not on 0.23 mapred.output.compress=true mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec -- settings that work on 0.23 mapreduce.output.fileoutputformat.compress=true mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec This breaks backwards compatibility and causes existing jobs to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3596:
---
Attachment: MAPREDUCE-3596-20120112.txt
Addressing review comment.
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.txt, logs.tar.bz2,
> logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3596:
---
Status: Open (was: Patch Available)
Thanks for the review Robert.
Makes sense, updating the patch.
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.txt, logs.tar.bz2,
> logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185155#comment-13185155 ] Jitendra Nath Pandey commented on MAPREDUCE-3664: - Committed. Thanks to Brandon Li! > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185148#comment-13185148 ] Brandon Li commented on MAPREDUCE-3664: --- move this defect from HDFS to mapreduce since this apt.vm file is in mapreduce site. > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li moved HDFS-2778 to MAPREDUCE-3664: - Component/s: (was: documentation) documentation Assignee: (was: Brandon Li) Target Version/s: (was: 0.23.1, 0.24.0) Affects Version/s: (was: 0.24.0) (was: 0.23.0) 0.24.0 0.23.0 Key: MAPREDUCE-3664 (was: HDFS-2778) Project: Hadoop Map/Reduce (was: Hadoop HDFS) > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3397) Support no sort dataflow in map output and reduce merge phrase
[ https://issues.apache.org/jira/browse/MAPREDUCE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185110#comment-13185110 ] Robert Joseph Evans commented on MAPREDUCE-3397: I have a couple of comments about the patch. NOTE I have just done a quick pass and have not looked in depth at the code. There are several places that I want to pick apart before giving my approval. # Combiners are not compatible with mapred.map.output.sort. Is there a reason why we could not make combiners work with this, so long as they must follow the same assumption that they will not get sorted input? If the algorithms you are thinking about would never get any benefit from a combiner, could you also add the check in the client. I would much rather have the client blow up with an error instead of waiting for my map tasks to launch and then blow up 4+ times before I get the error. # In your test you never validate that the output is what you expected it to be. That may be hard as it may not be deterministic because there is no sorting, but it would be nice to have something verify that the code did work as expected. Not just that it did not crash. # mapred-default.xml Please add mapred.map.output.sort to mapred-default.xml. Include with it a brief explanation of what it does. # There is no documentation or examples. This is a new feature that could be very useful to lots of people, but if they never know it is there it will not be used. Could you include in your patch updates to the documentation about how to use this, and some useful examples, preferably simple. Perhaps an example computing CTR would be nice. # Performance. The entire reason for this change is to improve performance, but I have not seen any numbers showing a performance improvement. No numbers at all in fact. It would be great if you could include here some numbers along with the code you used for your benchmark and a description of your setup. I have spent time on different performance teams, and performance improvement efforts from a huge search engine to an OS on a cell phone and the one thing I have learned is that you have to go off of the numbers because well at least for me my intuition is often wrong and what I thought would make it faster slowed it down instead. # Trunk. This patch is specific to 0.20/1.0 line. Before this can get merged into the 0.20/1.0 lines we really need an equivalent patch for trunk, and possibly 0.21, 0.22, and 0.23. This is so there are no regressions. It may be a while off after you get the 1.0 patch cleaned up though. > Support no sort dataflow in map output and reduce merge phrase > -- > > Key: MAPREDUCE-3397 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3397 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task >Affects Versions: 0.20.205.0 >Reporter: Binglin Chang >Assignee: Binglin Chang > Attachments: MAPREDUCE-3397-nosort.v1.patch > > > In our experience, many data aggregation style queries/jobs don't need to > sort the intermediate data. In fact reducer side can use hashmap or even > array to do application level aggregations. For example, consider computing > CTR using display log & click log in sponsored search. Map side just emit > (adv_id, clk_cnt, dis_cnt), reduce side aggregate clk_cnt and dis_cnt for > every adv_id, cause adv_id is integer, we can partition adv_id by range: > ** reduce0: 0-10 > ** reduce1: 10-20 > ** ... > ** reduceM: xxx-max adv-id > Then the reducer can use an array(for example: int [100][2]) to store the > aggregated clk_cnt & dis_cnt, and we don't need the framework to sort > intermediate data anymore. > By supporting no sort, we can gain a lot of performance improvements: > # Eliminate map side sort & merge. > KV paris need to sort by partition first, but this can be done using a > liner time counting sort, which is much faster than quick sort. > Just merge spill segments one by one, doesn't need to use heap merge. > # Eliminate shuffle phrase barrier, reducer can start to processing data > before all map output data are copied & merged. > For most cases, memory won't be a problem, cause keys are divided to many > partitions, each reducers only process a small subset of the global key set. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185057#comment-13185057
]
Robert Joseph Evans commented on MAPREDUCE-3596:
+1 (non-binding) I just have one comment and it is very minor, so feel free to
ignore it, my +1 stands with or without it.
In RMNodeImpl getAppsToCleanup and getContainersToCleanUp. It would probably
be slightly more efficient to do something like {code}
this.readLock.lock();
try {
return new ArrayList(this.containersToClean);
} finally {
this.readLock.unlock();
}{code} It will ensure that for even large arrays internally in ArrayList only
one array will need to be allocated.
> Sort benchmark got hang after completion of 99% map phase
> -
>
> Key: MAPREDUCE-3596
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3596-20120111.1.txt,
> MAPREDUCE-3596-20120111.txt, logs.tar.bz2, logs.tar.bz2
>
>
> Courtesy [~vinaythota]
> {quote}
> Ran sort benchmark couple of times and every time the job got hang after
> completion 99% map phase. There are some map tasks failed. Also it's not
> scheduled some of the pending map tasks.
> Cluster size is 350 nodes.
> Build Details:
> ==
> Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
> branches/branch-0.23/hadoop-common-project/hadoop-common
> ResourceManager version:revision 1212681 by someone source checksum
> on Fri Dec 9 16:52:07 PST 2011
> Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
> 2011
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3663) After submitting a job. If the Runjar process gets killed ,then the job is hanging
After submitting a job. If the Runjar process gets killed ,then the job is hanging -- Key: MAPREDUCE-3663 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3663 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramgopal N When the job is submitted...Runjar process is created and the YarnChild processes also start running.If at this time ,the RunJar process is getting killed, the job is hanging. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3648) TestJobConf failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184947#comment-13184947 ] Hudson commented on MAPREDUCE-3648: --- Integrated in Hadoop-Mapreduce-trunk #956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/]) MAPREDUCE-3648. TestJobConf failing. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230202 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobConf.java > TestJobConf failing > --- > > Key: MAPREDUCE-3648 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3648 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3648.patch, MAPREDUCE-3648.patch > > > TestJobConf is failing: > testFindContainingJar > testFindContainingJarWithPlus > java.lang.ClassNotFoundException: ClassWithNoPackage > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.mapred.TestJobConf.testJarAtPath(TestJobConf.java:78) > at > org.apache.hadoop.mapred.TestJobConf.testFindContainingJar(TestJobConf.java:44) > Looks like perhaps a classpath issue. > TestQueueManagerRefresh also has failures and I'm wondering might be related > as it doesn't seem to pick up a config file written out to > build/test/extraconf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184946#comment-13184946 ] Hudson commented on MAPREDUCE-3512: --- Integrated in Hadoop-Mapreduce-trunk #956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/]) MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230353 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3652) org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184943#comment-13184943 ] Hudson commented on MAPREDUCE-3652: --- Integrated in Hadoop-Mapreduce-trunk #956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/]) MAPREDUCE-3652. org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230328 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskFail.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestWebUIAuthorization.java > org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails > > > Key: MAPREDUCE-3652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3652 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3652.patch, MAPREDUCE-3652.patch, > MAPREDUCE-3652.patch > > > org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails. > This is testing the old jsp web interfaces. I think this test should just be > removed. > Any objections? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184944#comment-13184944 ] Hudson commented on MAPREDUCE-3625: --- Integrated in Hadoop-Mapreduce-trunk #956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/]) MAPREDUCE-3625. CapacityScheduler web-ui display of queue's used capacity is broken. (Jason Lowe via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230336 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > CapacityScheduler web-ui display of queue's used capacity is broken > --- > > Key: MAPREDUCE-3625 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.1 > > Attachments: CapacityScheduler.png, MAPREDUCE-3625.patch > > > The display of the queue's used capacity at runtime is broken because it > display's 'used' relative to the queue's capacity and not the parent's > capacity as shown in the above attachment. > The display should be relative to parent's capacity and not leaf queues as > everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3651) TestQueueManagerRefresh fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184942#comment-13184942 ] Hudson commented on MAPREDUCE-3651: --- Integrated in Hadoop-Mapreduce-trunk #956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/]) MAPREDUCE-3651. TestQueueManagerRefresh fails. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230292 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java > TestQueueManagerRefresh fails > - > > Key: MAPREDUCE-3651 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3651 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3651.patch, MAPREDUCE-3651.patch > > > The following tests fail: > org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshWithRemovedQueues > org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshOfSchedulerProperties > > It looks like its simply trying to remove one of the queues but the remove is > failing.It looks like MAPREDUCE-3328. mapred queue -list output inconsistent > and missing child queues - change the getChilren routine to do a new > JobQueueInfo on each one when returning it which is making the remove routine > fail since they aren't the same object now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3553) Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
[ https://issues.apache.org/jira/browse/MAPREDUCE-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184939#comment-13184939 ] Hudson commented on MAPREDUCE-3553: --- Integrated in Hadoop-Mapreduce-trunk #956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/]) MAPREDUCE-3553. Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/GenericExceptionHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/RemoteExceptionData.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java > Add support for data returned when exceptions thrown from web service apis to > be in either xml or in JSON > - > > Key: MAPREDUCE-3553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3553 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Minor > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3553.patch, MAPREDUCE-3553.patch, > MAPREDUCE-3553.patch > > > When the web services apis for rm, nm, app master, and job history server > throw an exception - like bad request, not found, they always return the data > in JSON format. It would be nice to return based on what they requested - > xml or JSON. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3645) TestJobHistory fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184938#comment-13184938 ] Hudson commented on MAPREDUCE-3645: --- Integrated in Hadoop-Mapreduce-trunk #956 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/956/]) MAPREDUCE-3645. TestJobHistory fails. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230307 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobHistory.java > TestJobHistory fails > > > Key: MAPREDUCE-3645 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3645 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3645.patch > > > TestJobHistory fails. > >>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS > >>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem > >>> > >>> org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS > >>> org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile > It looks like this was introduced by MAPREDUCE-3349 and the issue is that the > test expects the hostname to be in the format rackname/hostname, but with > 3349 it split those apart into 2 different fields. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184935#comment-13184935 ] Hudson commented on MAPREDUCE-3512: --- Integrated in Hadoop-Mapreduce-0.23-Build #158 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/158/]) MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth. svn merge --ignore-ancestry -c 1230353 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230354 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3648) TestJobConf failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184936#comment-13184936 ] Hudson commented on MAPREDUCE-3648: --- Integrated in Hadoop-Mapreduce-0.23-Build #158 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/158/]) MAPREDUCE-3648. TestJobConf failing. (Thomas Graves via mahadev) - Merging r1230202 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230203 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobConf.java > TestJobConf failing > --- > > Key: MAPREDUCE-3648 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3648 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3648.patch, MAPREDUCE-3648.patch > > > TestJobConf is failing: > testFindContainingJar > testFindContainingJarWithPlus > java.lang.ClassNotFoundException: ClassWithNoPackage > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.mapred.TestJobConf.testJarAtPath(TestJobConf.java:78) > at > org.apache.hadoop.mapred.TestJobConf.testFindContainingJar(TestJobConf.java:44) > Looks like perhaps a classpath issue. > TestQueueManagerRefresh also has failures and I'm wondering might be related > as it doesn't seem to pick up a config file written out to > build/test/extraconf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3553) Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
[ https://issues.apache.org/jira/browse/MAPREDUCE-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184929#comment-13184929 ] Hudson commented on MAPREDUCE-3553: --- Integrated in Hadoop-Mapreduce-0.23-Build #158 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/158/]) MAPREDUCE-3553. Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON. (Thomas Graves via mahadev) - Merging r1230330 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230333 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/JAXBContextResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/GenericExceptionHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/RemoteExceptionData.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/JAXBContextResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java > Add support for data returned when exceptions thrown from web service apis to > be in either xml or in JSON > - > > Key: MAPREDUCE-3553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3553 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Minor > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3553.patch, MAPREDUCE-3553.patch, > MAPREDUCE-3553.patch > > > When the web services apis for rm, nm, app master, and job history server > throw an exception - like bad request, not found, they always return the data > in JSON format. It would be nice to return based on what they requested - > xml or JSON. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184933#comment-13184933 ] Hudson commented on MAPREDUCE-3625: --- Integrated in Hadoop-Mapreduce-0.23-Build #158 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/158/]) MAPREDUCE-3625. CapacityScheduler web-ui display of queue's used capacity is broken. (Jason Lowe via mahadev) - Merging r1230336 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230338 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > CapacityScheduler web-ui display of queue's used capacity is broken > --- > > Key: MAPREDUCE-3625 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.1 > > Attachments: CapacityScheduler.png, MAPREDUCE-3625.patch > > > The display of the queue's used capacity at runtime is broken because it > display's 'used' relative to the queue's capacity and not the parent's > capacity as shown in the above attachment. > The display should be relative to parent's capacity and not leaf queues as > everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3652) org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184932#comment-13184932 ] Hudson commented on MAPREDUCE-3652: --- Integrated in Hadoop-Mapreduce-0.23-Build #158 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/158/]) MAPREDUCE-3652. org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails. (Thomas Graves via mahadev) - Merging r1230328 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230329 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskFail.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestWebUIAuthorization.java > org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails > > > Key: MAPREDUCE-3652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3652 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3652.patch, MAPREDUCE-3652.patch, > MAPREDUCE-3652.patch > > > org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails. > This is testing the old jsp web interfaces. I think this test should just be > removed. > Any objections? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3651) TestQueueManagerRefresh fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184931#comment-13184931 ] Hudson commented on MAPREDUCE-3651: --- Integrated in Hadoop-Mapreduce-0.23-Build #158 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/158/]) MAPREDUCE-3651. TestQueueManagerRefresh fails. (Thomas Graves via mahadev) - Merging r1230292 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230295 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java > TestQueueManagerRefresh fails > - > > Key: MAPREDUCE-3651 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3651 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3651.patch, MAPREDUCE-3651.patch > > > The following tests fail: > org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshWithRemovedQueues > org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshOfSchedulerProperties > > It looks like its simply trying to remove one of the queues but the remove is > failing.It looks like MAPREDUCE-3328. mapred queue -list output inconsistent > and missing child queues - change the getChilren routine to do a new > JobQueueInfo on each one when returning it which is making the remove routine > fail since they aren't the same object now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3645) TestJobHistory fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184928#comment-13184928 ] Hudson commented on MAPREDUCE-3645: --- Integrated in Hadoop-Mapreduce-0.23-Build #158 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/158/]) MAPREDUCE-3645. TestJobHistory fails. (Thomas Graves via mahadev) - Merging r1230307 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230308 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobHistory.java > TestJobHistory fails > > > Key: MAPREDUCE-3645 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3645 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3645.patch > > > TestJobHistory fails. > >>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS > >>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem > >>> > >>> org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS > >>> org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile > It looks like this was introduced by MAPREDUCE-3349 and the issue is that the > test expects the hostname to be in the format rackname/hostname, but with > 3349 it split those apart into 2 different fields. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184910#comment-13184910 ] Hudson commented on MAPREDUCE-3625: --- Integrated in Hadoop-Hdfs-0.23-Build #136 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/136/]) MAPREDUCE-3625. CapacityScheduler web-ui display of queue's used capacity is broken. (Jason Lowe via mahadev) - Merging r1230336 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230338 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > CapacityScheduler web-ui display of queue's used capacity is broken > --- > > Key: MAPREDUCE-3625 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.1 > > Attachments: CapacityScheduler.png, MAPREDUCE-3625.patch > > > The display of the queue's used capacity at runtime is broken because it > display's 'used' relative to the queue's capacity and not the parent's > capacity as shown in the above attachment. > The display should be relative to parent's capacity and not leaf queues as > everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3648) TestJobConf failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184913#comment-13184913 ] Hudson commented on MAPREDUCE-3648: --- Integrated in Hadoop-Hdfs-0.23-Build #136 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/136/]) MAPREDUCE-3648. TestJobConf failing. (Thomas Graves via mahadev) - Merging r1230202 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230203 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobConf.java > TestJobConf failing > --- > > Key: MAPREDUCE-3648 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3648 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3648.patch, MAPREDUCE-3648.patch > > > TestJobConf is failing: > testFindContainingJar > testFindContainingJarWithPlus > java.lang.ClassNotFoundException: ClassWithNoPackage > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.mapred.TestJobConf.testJarAtPath(TestJobConf.java:78) > at > org.apache.hadoop.mapred.TestJobConf.testFindContainingJar(TestJobConf.java:44) > Looks like perhaps a classpath issue. > TestQueueManagerRefresh also has failures and I'm wondering might be related > as it doesn't seem to pick up a config file written out to > build/test/extraconf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184912#comment-13184912 ] Hudson commented on MAPREDUCE-3512: --- Integrated in Hadoop-Hdfs-0.23-Build #136 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/136/]) MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth. svn merge --ignore-ancestry -c 1230353 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230354 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3652) org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184909#comment-13184909 ] Hudson commented on MAPREDUCE-3652: --- Integrated in Hadoop-Hdfs-0.23-Build #136 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/136/]) MAPREDUCE-3652. org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails. (Thomas Graves via mahadev) - Merging r1230328 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230329 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskFail.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestWebUIAuthorization.java > org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails > > > Key: MAPREDUCE-3652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3652 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3652.patch, MAPREDUCE-3652.patch, > MAPREDUCE-3652.patch > > > org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails. > This is testing the old jsp web interfaces. I think this test should just be > removed. > Any objections? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3651) TestQueueManagerRefresh fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184908#comment-13184908 ] Hudson commented on MAPREDUCE-3651: --- Integrated in Hadoop-Hdfs-0.23-Build #136 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/136/]) MAPREDUCE-3651. TestQueueManagerRefresh fails. (Thomas Graves via mahadev) - Merging r1230292 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230295 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java > TestQueueManagerRefresh fails > - > > Key: MAPREDUCE-3651 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3651 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3651.patch, MAPREDUCE-3651.patch > > > The following tests fail: > org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshWithRemovedQueues > org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshOfSchedulerProperties > > It looks like its simply trying to remove one of the queues but the remove is > failing.It looks like MAPREDUCE-3328. mapred queue -list output inconsistent > and missing child queues - change the getChilren routine to do a new > JobQueueInfo on each one when returning it which is making the remove routine > fail since they aren't the same object now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3645) TestJobHistory fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184905#comment-13184905 ] Hudson commented on MAPREDUCE-3645: --- Integrated in Hadoop-Hdfs-0.23-Build #136 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/136/]) MAPREDUCE-3645. TestJobHistory fails. (Thomas Graves via mahadev) - Merging r1230307 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230308 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobHistory.java > TestJobHistory fails > > > Key: MAPREDUCE-3645 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3645 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3645.patch > > > TestJobHistory fails. > >>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS > >>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem > >>> > >>> org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS > >>> org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile > It looks like this was introduced by MAPREDUCE-3349 and the issue is that the > test expects the hostname to be in the format rackname/hostname, but with > 3349 it split those apart into 2 different fields. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3553) Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
[ https://issues.apache.org/jira/browse/MAPREDUCE-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184906#comment-13184906 ] Hudson commented on MAPREDUCE-3553: --- Integrated in Hadoop-Hdfs-0.23-Build #136 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/136/]) MAPREDUCE-3553. Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON. (Thomas Graves via mahadev) - Merging r1230330 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230333 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/JAXBContextResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/GenericExceptionHandler.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/RemoteExceptionData.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/JAXBContextResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java > Add support for data returned when exceptions thrown from web service apis to > be in either xml or in JSON > - > > Key: MAPREDUCE-3553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3553 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Minor > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3553.patch, MAPREDUCE-3553.patch, > MAPREDUCE-3553.patch > > > When the web services apis for rm, nm, app master, and job history server > throw an exception - like bad request, not found, they always return the data > in JSON format. It would be nice to return based on what they requested - > xml or JSON. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184900#comment-13184900 ] Hudson commented on MAPREDUCE-3512: --- Integrated in Hadoop-Hdfs-trunk #923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/]) MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Contributed by Siddarth Seth. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230353 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3625) CapacityScheduler web-ui display of queue's used capacity is broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184898#comment-13184898 ] Hudson commented on MAPREDUCE-3625: --- Integrated in Hadoop-Hdfs-trunk #923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/]) MAPREDUCE-3625. CapacityScheduler web-ui display of queue's used capacity is broken. (Jason Lowe via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230336 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > CapacityScheduler web-ui display of queue's used capacity is broken > --- > > Key: MAPREDUCE-3625 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3625 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.1 > > Attachments: CapacityScheduler.png, MAPREDUCE-3625.patch > > > The display of the queue's used capacity at runtime is broken because it > display's 'used' relative to the queue's capacity and not the parent's > capacity as shown in the above attachment. > The display should be relative to parent's capacity and not leaf queues as > everything else in the display is relative to parent's capacity. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3648) TestJobConf failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184901#comment-13184901 ] Hudson commented on MAPREDUCE-3648: --- Integrated in Hadoop-Hdfs-trunk #923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/]) MAPREDUCE-3648. TestJobConf failing. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230202 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobConf.java > TestJobConf failing > --- > > Key: MAPREDUCE-3648 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3648 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3648.patch, MAPREDUCE-3648.patch > > > TestJobConf is failing: > testFindContainingJar > testFindContainingJarWithPlus > java.lang.ClassNotFoundException: ClassWithNoPackage > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.mapred.TestJobConf.testJarAtPath(TestJobConf.java:78) > at > org.apache.hadoop.mapred.TestJobConf.testFindContainingJar(TestJobConf.java:44) > Looks like perhaps a classpath issue. > TestQueueManagerRefresh also has failures and I'm wondering might be related > as it doesn't seem to pick up a config file written out to > build/test/extraconf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3652) org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184897#comment-13184897 ] Hudson commented on MAPREDUCE-3652: --- Integrated in Hadoop-Hdfs-trunk #923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/]) MAPREDUCE-3652. org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230328 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestTaskFail.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestWebUIAuthorization.java > org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails > > > Key: MAPREDUCE-3652 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3652 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3652.patch, MAPREDUCE-3652.patch, > MAPREDUCE-3652.patch > > > org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails. > This is testing the old jsp web interfaces. I think this test should just be > removed. > Any objections? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3651) TestQueueManagerRefresh fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184896#comment-13184896 ] Hudson commented on MAPREDUCE-3651: --- Integrated in Hadoop-Hdfs-trunk #923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/]) MAPREDUCE-3651. TestQueueManagerRefresh fails. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230292 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java > TestQueueManagerRefresh fails > - > > Key: MAPREDUCE-3651 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3651 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3651.patch, MAPREDUCE-3651.patch > > > The following tests fail: > org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshWithRemovedQueues > org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshOfSchedulerProperties > > It looks like its simply trying to remove one of the queues but the remove is > failing.It looks like MAPREDUCE-3328. mapred queue -list output inconsistent > and missing child queues - change the getChilren routine to do a new > JobQueueInfo on each one when returning it which is making the remove routine > fail since they aren't the same object now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3645) TestJobHistory fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184892#comment-13184892 ] Hudson commented on MAPREDUCE-3645: --- Integrated in Hadoop-Hdfs-trunk #923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/]) MAPREDUCE-3645. TestJobHistory fails. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230307 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobHistory.java > TestJobHistory fails > > > Key: MAPREDUCE-3645 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3645 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3645.patch > > > TestJobHistory fails. > >>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS > >>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem > >>> > >>> org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS > >>> org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile > It looks like this was introduced by MAPREDUCE-3349 and the issue is that the > test expects the hostname to be in the format rackname/hostname, but with > 3349 it split those apart into 2 different fields. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3553) Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
[ https://issues.apache.org/jira/browse/MAPREDUCE-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184893#comment-13184893 ] Hudson commented on MAPREDUCE-3553: --- Integrated in Hadoop-Hdfs-trunk #923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/923/]) MAPREDUCE-3553. Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1230330 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/GenericExceptionHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/RemoteExceptionData.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServicesApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java > Add support for data returned when exceptions thrown from web service apis to > be in either xml or in JSON > - > > Key: MAPREDUCE-3553 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3553 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Minor > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3553.patch, MAPREDUCE-3553.patch, > MAPREDUCE-3553.patch > > > When the web services apis for rm, nm, app master, and job history server > throw an exception - like bad request, not found, they always return the data > in JSON format. It would be nice to return based on what they requested - > xml or JSON. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3582) Move successfully passing MR1 tests to MR2 maven tree.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184867#comment-13184867 ] Hadoop QA commented on MAPREDUCE-3582: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510357/MAPREDUCE-3582-whole_rev7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1086 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1605//console This message is automatically generated. > Move successfully passing MR1 tests to MR2 maven tree. > -- > > Key: MAPREDUCE-3582 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3582 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3582-part1_rev3.patch, > MAPREDUCE-3582-part1_rev4.patch, MAPREDUCE-3582-part1_rev5.patch, > MAPREDUCE-3582-part1_rev6.patch, MAPREDUCE-3582-part1_rev7.patch, > MAPREDUCE-3582-part2_rev3.patch, MAPREDUCE-3582-part2_rev4.patch, > MAPREDUCE-3582-part2_rev5.patch, MAPREDUCE-3582-part2_rev6.patch, > MAPREDUCE-3582-part2_rev7.patch, MAPREDUCE-3582-whole_rev3.patch, > MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev4.patch, > MAPREDUCE-3582-whole_rev5.patch, MAPREDUCE-3582-whole_rev6.patch, > MAPREDUCE-3582-whole_rev7.patch, MAPREDUCE-3582.patch, > MAPREDUCE-3582_rev2.patch, mv_script_MR-3582.sh, mv_script_MR-3582_rev2.sh, > mv_script_MR-3582_rev3.sh, mv_script_MR-3582_rev4.sh, > mv_script_MR-3582_rev5.sh, mv_script_MR-3582_rev6.sh, > mv_script_MR-3582_rev7.sh > > > This ticket will track moving mr1 tests that are passing successfully to mr2 > maven tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3582) Move successfully passing MR1 tests to MR2 maven tree.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184866#comment-13184866 ] Hadoop QA commented on MAPREDUCE-3582: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510356/mv_script_MR-3582_rev7.sh against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 259 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1604//console This message is automatically generated. > Move successfully passing MR1 tests to MR2 maven tree. > -- > > Key: MAPREDUCE-3582 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3582 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3582-part1_rev3.patch, > MAPREDUCE-3582-part1_rev4.patch, MAPREDUCE-3582-part1_rev5.patch, > MAPREDUCE-3582-part1_rev6.patch, MAPREDUCE-3582-part1_rev7.patch, > MAPREDUCE-3582-part2_rev3.patch, MAPREDUCE-3582-part2_rev4.patch, > MAPREDUCE-3582-part2_rev5.patch, MAPREDUCE-3582-part2_rev6.patch, > MAPREDUCE-3582-part2_rev7.patch, MAPREDUCE-3582-whole_rev3.patch, > MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev4.patch, > MAPREDUCE-3582-whole_rev5.patch, MAPREDUCE-3582-whole_rev6.patch, > MAPREDUCE-3582-whole_rev7.patch, MAPREDUCE-3582.patch, > MAPREDUCE-3582_rev2.patch, mv_script_MR-3582.sh, mv_script_MR-3582_rev2.sh, > mv_script_MR-3582_rev3.sh, mv_script_MR-3582_rev4.sh, > mv_script_MR-3582_rev5.sh, mv_script_MR-3582_rev6.sh, > mv_script_MR-3582_rev7.sh > > > This ticket will track moving mr1 tests that are passing successfully to mr2 > maven tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3582) Move successfully passing MR1 tests to MR2 maven tree.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-3582: Attachment: MAPREDUCE-3582-whole_rev7.patch Attaching the updated whole patch for jenkins. > Move successfully passing MR1 tests to MR2 maven tree. > -- > > Key: MAPREDUCE-3582 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3582 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3582-part1_rev3.patch, > MAPREDUCE-3582-part1_rev4.patch, MAPREDUCE-3582-part1_rev5.patch, > MAPREDUCE-3582-part1_rev6.patch, MAPREDUCE-3582-part1_rev7.patch, > MAPREDUCE-3582-part2_rev3.patch, MAPREDUCE-3582-part2_rev4.patch, > MAPREDUCE-3582-part2_rev5.patch, MAPREDUCE-3582-part2_rev6.patch, > MAPREDUCE-3582-part2_rev7.patch, MAPREDUCE-3582-whole_rev3.patch, > MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev4.patch, > MAPREDUCE-3582-whole_rev5.patch, MAPREDUCE-3582-whole_rev6.patch, > MAPREDUCE-3582-whole_rev7.patch, MAPREDUCE-3582.patch, > MAPREDUCE-3582_rev2.patch, mv_script_MR-3582.sh, mv_script_MR-3582_rev2.sh, > mv_script_MR-3582_rev3.sh, mv_script_MR-3582_rev4.sh, > mv_script_MR-3582_rev5.sh, mv_script_MR-3582_rev6.sh, > mv_script_MR-3582_rev7.sh > > > This ticket will track moving mr1 tests that are passing successfully to mr2 > maven tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3582) Move successfully passing MR1 tests to MR2 maven tree.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-3582: Attachment: mv_script_MR-3582_rev7.sh MAPREDUCE-3582-part1_rev7.patch MAPREDUCE-3582-part2_rev7.patch Thanks Tucu, I am attaching the updated patch addressing all comments. > Move successfully passing MR1 tests to MR2 maven tree. > -- > > Key: MAPREDUCE-3582 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3582 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-3582-part1_rev3.patch, > MAPREDUCE-3582-part1_rev4.patch, MAPREDUCE-3582-part1_rev5.patch, > MAPREDUCE-3582-part1_rev6.patch, MAPREDUCE-3582-part1_rev7.patch, > MAPREDUCE-3582-part2_rev3.patch, MAPREDUCE-3582-part2_rev4.patch, > MAPREDUCE-3582-part2_rev5.patch, MAPREDUCE-3582-part2_rev6.patch, > MAPREDUCE-3582-part2_rev7.patch, MAPREDUCE-3582-whole_rev3.patch, > MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev4.patch, > MAPREDUCE-3582-whole_rev5.patch, MAPREDUCE-3582-whole_rev6.patch, > MAPREDUCE-3582.patch, MAPREDUCE-3582_rev2.patch, mv_script_MR-3582.sh, > mv_script_MR-3582_rev2.sh, mv_script_MR-3582_rev3.sh, > mv_script_MR-3582_rev4.sh, mv_script_MR-3582_rev5.sh, > mv_script_MR-3582_rev6.sh, mv_script_MR-3582_rev7.sh > > > This ticket will track moving mr1 tests that are passing successfully to mr2 > maven tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3532) When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-3532:
--
Component/s: mrv2
> When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs
> webserver component picks up random port, NM keeps on Reporting 0 port to RM
> --
>
> Key: MAPREDUCE-3532
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3532
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
>Affects Versions: 0.23.1
>Reporter: Karam Singh
>Priority: Critical
> Attachments: MAPREDUCE-3532-1.patch, MAPREDUCE-3532.patch
>
>
> I tried following -:
> yarn.nodemanager.address=0.0.0.0:0
> yarn.nodemanager.webapp.address=0.0.0.0:0
> yarn.nodemanager.localizer.address=0.0.0.0:0
> mapreduce.shuffle.port=0
> When 0 is provided as number in yarn.nodemanager.webapp.address.
> NM instantiate WebServer as 0 piort e.g.
> {code}
> 2011-12-08 11:33:02,467 INFO
> org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
> NMWebApp at 0.0.0.0:0
> {code}
> After that WebServer pick up some random port e.g.
> {code}
> 2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> to port 36272
> 2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
> 2011-12-08 11:33:02,831 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:36272
> 2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
> /node started at 36272
> {code}
> And NM WebServer responds correctly but
> RM's cluster/Nodes page shows the following -:
> {code}
> /Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
> {code}
> Whereas NM:0 is not clickable.
> Seems even NM's webserver pick random port but it never gets updated and so
> NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
> But verified that MR job runs successfully with random.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
