[jira] [Commented] (MAPREDUCE-3532) When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186187#comment-13186187
]
Hudson commented on MAPREDUCE-3532:
---
Integrated in Hadoop-Hdfs-trunk #925 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/925/])
MAPREDUCE-3532. Modified NM to report correct http address when an
ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231342
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs
webserver component picks up random port, NM keeps on Reporting 0 port to RM
--
Key: MAPREDUCE-3532
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3532
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3532-1.patch, MAPREDUCE-3532.patch
I tried following -:
yarn.nodemanager.address=0.0.0.0:0
yarn.nodemanager.webapp.address=0.0.0.0:0
yarn.nodemanager.localizer.address=0.0.0.0:0
mapreduce.shuffle.port=0
When 0 is provided as number in yarn.nodemanager.webapp.address.
NM instantiate WebServer as 0 piort e.g.
{code}
2011-12-08 11:33:02,467 INFO
org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
NMWebApp at 0.0.0.0:0
{code}
After that WebServer pick up some random port e.g.
{code}
2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 36272
2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
2011-12-08 11:33:02,831 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:36272
2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/node started at 36272
{code}
And NM WebServer responds correctly but
RM's cluster/Nodes page shows the following -:
{code}
/Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
{code}
Whereas NM:0 is not clickable.
Seems even NM's webserver pick random port but it never gets updated and so
NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
But verified that MR job runs successfully with random.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186189#comment-13186189
]
Hudson commented on MAPREDUCE-3656:
---
Integrated in Hadoop-Hdfs-trunk #925 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/925/])
MAPREDUCE-3656. Fixed a race condition in MR AM which is failing the sort
benchmark consistently. Contributed by Siddarth Seth.
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231314
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptListener.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
Sort job on 350 scale is consistently failing with latest MRV2 code
Key: MAPREDUCE-3656
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2, resourcemanager
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Siddharth Seth
Priority: Blocker
Fix For: 0.23.1
Attachments: MR3656.txt, MR3656.txt, MR3656.txt
With the code checked out on last two days.
Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
failing for around last 6 runs
When around 50% of maps are completed, suddenly job jumps to failed state.
On looking at NM log, found RM sent Stop Container Request to NM for AM
container.
But at INFO level from RM log not able find why RM is killing AM when job is
not killed manually.
One thing found common on failed AM logs is -:
org.apache.hadoop.yarn.state.InvalidStateTransitonException
With with different.
For e.g. One log says -:
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
TA_UPDATE at ASSIGNED
{code}
Whereas other logs says -:
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_COUNTER_UPDATE at ERROR
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186188#comment-13186188
]
Hudson commented on MAPREDUCE-3596:
---
Integrated in Hadoop-Hdfs-trunk #925 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/925/])
MAPREDUCE-3596. Fix scheduler to handle cleaned up containers, which NMs
may subsequently report as running. (Contributed by Vinod Kumar Vavilapalli)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231297
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
Sort benchmark got hang after completion of 99% map phase
-
Key: MAPREDUCE-3596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.23.1
Attachments: MAPREDUCE-3596-20120111.1.txt,
MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.1.txt,
MAPREDUCE-3596-20120112.txt, logs.tar.bz2, logs.tar.bz2
Courtesy [~vinaythota]
{quote}
Ran sort benchmark couple of times and every time the job got hang after
completion 99% map phase. There are some map tasks failed. Also it's not
scheduled some of the pending map tasks.
Cluster size is 350 nodes.
Build Details:
==
Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
branches/branch-0.23/hadoop-common-project/hadoop-common
ResourceManager version:revision 1212681 by someone source checksum
on Fri Dec 9 16:52:07 PST 2011
Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
2011
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3404) Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false
[ https://issues.apache.org/jira/browse/MAPREDUCE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186190#comment-13186190 ] Hudson commented on MAPREDUCE-3404: --- Integrated in Hadoop-Hdfs-trunk #925 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/925/]) MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231395 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false --- Key: MAPREDUCE-3404 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3404 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, mrv2 Affects Versions: 0.23.0 Environment: Hadoop version is: Hadoop 0.23.0.1110031628 10 node test cluster Reporter: patrick white Assignee: Eric Payne Priority: Critical Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3404.1.txt, MAPREDUCE-3404.2.txt When forcing a mapper to take significantly longer than other map tasks, speculative map tasks are launched even if the mapreduce.job.maps.speculative.execution parameter is set to 'false'. Testcase: ran default WordCount job with spec execution set to false for both map and reduce but still saw a fifth mapper task launch, ran job as follows: hadoop --config config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=false -Dmapreduce.job.reduces.speculative.execution=false /tmp/test_file_of_words* /tmp/file_of_words.out Input data was 4 text files hdfs blocksize, with same word pattern plus one diff text line in each file, fourth file was 4 times as large as others: hadoop --config config fs -ls /tmp Found 5 items drwxr-xr-x - user hdfs 0 2011-10-20 16:17 /tmp/file_of_words.out -rw-r--r-- 3 user hdfs 62800021 2011-10-20 14:45 /tmp/test_file_of_words1 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words2 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words3 -rw-r--r-- 3 user hdfs 271708312 2011-10-20 15:50 /tmp/test_file_of_words4 Job launched 5 mappers despite spec exec set to false, output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=273540 SLOTS_MILLIS_REDUCES=212876 Reran same case as above only set both spec exec params to 'true', same results only this time the fifth task being launched is expected since spec exec = true. job run: hadoop --config config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=true -Dmapreduce.job.reduces.speculative.execution=true /tmp/test_file_of_words* /tmp/file_of_words.out output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=279653 SLOTS_MILLIS_REDUCES=211474 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3532) When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186191#comment-13186191
]
Hudson commented on MAPREDUCE-3532:
---
Integrated in Hadoop-Hdfs-0.23-Build #138 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/138/])
MAPREDUCE-3532. Modified NM to report correct http address when an
ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.
svn merge --ignore-ancestry -c 1231342 ../../trunk/
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231344
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs
webserver component picks up random port, NM keeps on Reporting 0 port to RM
--
Key: MAPREDUCE-3532
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3532
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3532-1.patch, MAPREDUCE-3532.patch
I tried following -:
yarn.nodemanager.address=0.0.0.0:0
yarn.nodemanager.webapp.address=0.0.0.0:0
yarn.nodemanager.localizer.address=0.0.0.0:0
mapreduce.shuffle.port=0
When 0 is provided as number in yarn.nodemanager.webapp.address.
NM instantiate WebServer as 0 piort e.g.
{code}
2011-12-08 11:33:02,467 INFO
org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
NMWebApp at 0.0.0.0:0
{code}
After that WebServer pick up some random port e.g.
{code}
2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 36272
2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
2011-12-08 11:33:02,831 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:36272
2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/node started at 36272
{code}
And NM WebServer responds correctly but
RM's cluster/Nodes page shows the following -:
{code}
/Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
{code}
Whereas NM:0 is not clickable.
Seems even NM's webserver pick random port but it never gets updated and so
NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
But verified that MR job runs successfully with random.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186192#comment-13186192
]
Hudson commented on MAPREDUCE-3596:
---
Integrated in Hadoop-Hdfs-0.23-Build #138 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/138/])
merge MAPREDUCE-3596 from trunk. Fix scheduler to handle cleaned up
containers, which NMs may subsequently report as running. (Contributed by Vinod
Kumar Vavilapalli)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231303
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
Sort benchmark got hang after completion of 99% map phase
-
Key: MAPREDUCE-3596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.23.1
Attachments: MAPREDUCE-3596-20120111.1.txt,
MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.1.txt,
MAPREDUCE-3596-20120112.txt, logs.tar.bz2, logs.tar.bz2
Courtesy [~vinaythota]
{quote}
Ran sort benchmark couple of times and every time the job got hang after
completion 99% map phase. There are some map tasks failed. Also it's not
scheduled some of the pending map tasks.
Cluster size is 350 nodes.
Build Details:
==
Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
branches/branch-0.23/hadoop-common-project/hadoop-common
ResourceManager version:revision 1212681 by someone source checksum
on Fri Dec 9 16:52:07 PST 2011
Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
2011
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on
[jira] [Commented] (MAPREDUCE-3404) Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false
[ https://issues.apache.org/jira/browse/MAPREDUCE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186194#comment-13186194 ] Hudson commented on MAPREDUCE-3404: --- Integrated in Hadoop-Hdfs-0.23-Build #138 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/138/]) MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. svn merge --ignore-ancestry -c 1231395 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231397 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false --- Key: MAPREDUCE-3404 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3404 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, mrv2 Affects Versions: 0.23.0 Environment: Hadoop version is: Hadoop 0.23.0.1110031628 10 node test cluster Reporter: patrick white Assignee: Eric Payne Priority: Critical Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3404.1.txt, MAPREDUCE-3404.2.txt When forcing a mapper to take significantly longer than other map tasks, speculative map tasks are launched even if the mapreduce.job.maps.speculative.execution parameter is set to 'false'. Testcase: ran default WordCount job with spec execution set to false for both map and reduce but still saw a fifth mapper task launch, ran job as follows: hadoop --config config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=false -Dmapreduce.job.reduces.speculative.execution=false /tmp/test_file_of_words* /tmp/file_of_words.out Input data was 4 text files hdfs blocksize, with same word pattern plus one diff text line in each file, fourth file was 4 times as large as others: hadoop --config config fs -ls /tmp Found 5 items drwxr-xr-x - user hdfs 0 2011-10-20 16:17 /tmp/file_of_words.out -rw-r--r-- 3 user hdfs 62800021 2011-10-20 14:45 /tmp/test_file_of_words1 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words2 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words3 -rw-r--r-- 3 user hdfs 271708312 2011-10-20 15:50 /tmp/test_file_of_words4 Job launched 5 mappers despite spec exec set to false, output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=273540 SLOTS_MILLIS_REDUCES=212876 Reran same case as above only set both spec exec params to 'true', same results only this time the fifth task being launched is expected since spec exec = true. job run: hadoop --config config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=true -Dmapreduce.job.reduces.speculative.execution=true /tmp/test_file_of_words* /tmp/file_of_words.out output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=279653 SLOTS_MILLIS_REDUCES=211474 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186193#comment-13186193
]
Hudson commented on MAPREDUCE-3656:
---
Integrated in Hadoop-Hdfs-0.23-Build #138 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/138/])
MAPREDUCE-3656. Fixed a race condition in MR AM which is failing the sort
benchmark consistently. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1231314 ../../trunk/
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231316
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptListener.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
Sort job on 350 scale is consistently failing with latest MRV2 code
Key: MAPREDUCE-3656
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2, resourcemanager
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Siddharth Seth
Priority: Blocker
Fix For: 0.23.1
Attachments: MR3656.txt, MR3656.txt, MR3656.txt
With the code checked out on last two days.
Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
failing for around last 6 runs
When around 50% of maps are completed, suddenly job jumps to failed state.
On looking at NM log, found RM sent Stop Container Request to NM for AM
container.
But at INFO level from RM log not able find why RM is killing AM when job is
not killed manually.
One thing found common on failed AM logs is -:
org.apache.hadoop.yarn.state.InvalidStateTransitonException
With with different.
For e.g. One log says -:
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
TA_UPDATE at ASSIGNED
{code}
Whereas other logs says -:
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_COUNTER_UPDATE at ERROR
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186198#comment-13186198
]
Hudson commented on MAPREDUCE-3596:
---
Integrated in Hadoop-Mapreduce-0.23-Build #160 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/160/])
merge MAPREDUCE-3596 from trunk. Fix scheduler to handle cleaned up
containers, which NMs may subsequently report as running. (Contributed by Vinod
Kumar Vavilapalli)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231303
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
Sort benchmark got hang after completion of 99% map phase
-
Key: MAPREDUCE-3596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.23.1
Attachments: MAPREDUCE-3596-20120111.1.txt,
MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.1.txt,
MAPREDUCE-3596-20120112.txt, logs.tar.bz2, logs.tar.bz2
Courtesy [~vinaythota]
{quote}
Ran sort benchmark couple of times and every time the job got hang after
completion 99% map phase. There are some map tasks failed. Also it's not
scheduled some of the pending map tasks.
Cluster size is 350 nodes.
Build Details:
==
Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
branches/branch-0.23/hadoop-common-project/hadoop-common
ResourceManager version:revision 1212681 by someone source checksum
on Fri Dec 9 16:52:07 PST 2011
Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
2011
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more
[jira] [Commented] (MAPREDUCE-3532) When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186197#comment-13186197
]
Hudson commented on MAPREDUCE-3532:
---
Integrated in Hadoop-Mapreduce-0.23-Build #160 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/160/])
MAPREDUCE-3532. Modified NM to report correct http address when an
ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.
svn merge --ignore-ancestry -c 1231342 ../../trunk/
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231344
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs
webserver component picks up random port, NM keeps on Reporting 0 port to RM
--
Key: MAPREDUCE-3532
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3532
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3532-1.patch, MAPREDUCE-3532.patch
I tried following -:
yarn.nodemanager.address=0.0.0.0:0
yarn.nodemanager.webapp.address=0.0.0.0:0
yarn.nodemanager.localizer.address=0.0.0.0:0
mapreduce.shuffle.port=0
When 0 is provided as number in yarn.nodemanager.webapp.address.
NM instantiate WebServer as 0 piort e.g.
{code}
2011-12-08 11:33:02,467 INFO
org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
NMWebApp at 0.0.0.0:0
{code}
After that WebServer pick up some random port e.g.
{code}
2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 36272
2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
2011-12-08 11:33:02,831 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:36272
2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/node started at 36272
{code}
And NM WebServer responds correctly but
RM's cluster/Nodes page shows the following -:
{code}
/Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
{code}
Whereas NM:0 is not clickable.
Seems even NM's webserver pick random port but it never gets updated and so
NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
But verified that MR job runs successfully with random.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186199#comment-13186199
]
Hudson commented on MAPREDUCE-3656:
---
Integrated in Hadoop-Mapreduce-0.23-Build #160 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/160/])
MAPREDUCE-3656. Fixed a race condition in MR AM which is failing the sort
benchmark consistently. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1231314 ../../trunk/
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231316
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptListener.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
Sort job on 350 scale is consistently failing with latest MRV2 code
Key: MAPREDUCE-3656
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2, resourcemanager
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Siddharth Seth
Priority: Blocker
Fix For: 0.23.1
Attachments: MR3656.txt, MR3656.txt, MR3656.txt
With the code checked out on last two days.
Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
failing for around last 6 runs
When around 50% of maps are completed, suddenly job jumps to failed state.
On looking at NM log, found RM sent Stop Container Request to NM for AM
container.
But at INFO level from RM log not able find why RM is killing AM when job is
not killed manually.
One thing found common on failed AM logs is -:
org.apache.hadoop.yarn.state.InvalidStateTransitonException
With with different.
For e.g. One log says -:
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
TA_UPDATE at ASSIGNED
{code}
Whereas other logs says -:
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_COUNTER_UPDATE at ERROR
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3404) Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false
[ https://issues.apache.org/jira/browse/MAPREDUCE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186200#comment-13186200 ] Hudson commented on MAPREDUCE-3404: --- Integrated in Hadoop-Mapreduce-0.23-Build #160 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/160/]) MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. svn merge --ignore-ancestry -c 1231395 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231397 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false --- Key: MAPREDUCE-3404 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3404 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, mrv2 Affects Versions: 0.23.0 Environment: Hadoop version is: Hadoop 0.23.0.1110031628 10 node test cluster Reporter: patrick white Assignee: Eric Payne Priority: Critical Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3404.1.txt, MAPREDUCE-3404.2.txt When forcing a mapper to take significantly longer than other map tasks, speculative map tasks are launched even if the mapreduce.job.maps.speculative.execution parameter is set to 'false'. Testcase: ran default WordCount job with spec execution set to false for both map and reduce but still saw a fifth mapper task launch, ran job as follows: hadoop --config config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=false -Dmapreduce.job.reduces.speculative.execution=false /tmp/test_file_of_words* /tmp/file_of_words.out Input data was 4 text files hdfs blocksize, with same word pattern plus one diff text line in each file, fourth file was 4 times as large as others: hadoop --config config fs -ls /tmp Found 5 items drwxr-xr-x - user hdfs 0 2011-10-20 16:17 /tmp/file_of_words.out -rw-r--r-- 3 user hdfs 62800021 2011-10-20 14:45 /tmp/test_file_of_words1 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words2 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words3 -rw-r--r-- 3 user hdfs 271708312 2011-10-20 15:50 /tmp/test_file_of_words4 Job launched 5 mappers despite spec exec set to false, output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=273540 SLOTS_MILLIS_REDUCES=212876 Reran same case as above only set both spec exec params to 'true', same results only this time the fifth task being launched is expected since spec exec = true. job run: hadoop --config config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=true -Dmapreduce.job.reduces.speculative.execution=true /tmp/test_file_of_words* /tmp/file_of_words.out output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=279653 SLOTS_MILLIS_REDUCES=211474 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3532) When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-3532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186204#comment-13186204
]
Hudson commented on MAPREDUCE-3532:
---
Integrated in Hadoop-Mapreduce-trunk #958 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/958/])
MAPREDUCE-3532. Modified NM to report correct http address when an
ephemeral web port is configured. Contributed by Bhallamudi Venkata Siva Kamesh.
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231342
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServer.java
When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs
webserver component picks up random port, NM keeps on Reporting 0 port to RM
--
Key: MAPREDUCE-3532
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3532
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3532-1.patch, MAPREDUCE-3532.patch
I tried following -:
yarn.nodemanager.address=0.0.0.0:0
yarn.nodemanager.webapp.address=0.0.0.0:0
yarn.nodemanager.localizer.address=0.0.0.0:0
mapreduce.shuffle.port=0
When 0 is provided as number in yarn.nodemanager.webapp.address.
NM instantiate WebServer as 0 piort e.g.
{code}
2011-12-08 11:33:02,467 INFO
org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating
NMWebApp at 0.0.0.0:0
{code}
After that WebServer pick up some random port e.g.
{code}
2011-12-08 11:33:02,562 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 36272
2011-12-08 11:33:02,562 INFO org.mortbay.log: jetty-6.1.26
2011-12-08 11:33:02,831 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:36272
2011-12-08 11:33:02,831 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app
/node started at 36272
{code}
And NM WebServer responds correctly but
RM's cluster/Nodes page shows the following -:
{code}
/Rack RUNNING NM:57963 NM:0 Healthy 8-Dec-2011 11:33:01 Healthy 8 12 GB 0 KB
{code}
Whereas NM:0 is not clickable.
Seems even NM's webserver pick random port but it never gets updated and so
NM report 0 as HTTP port to RM causing NM Hyperlinks un-clickable
But verified that MR job runs successfully with random.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186205#comment-13186205
]
Hudson commented on MAPREDUCE-3596:
---
Integrated in Hadoop-Mapreduce-trunk #958 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/958/])
MAPREDUCE-3596. Fix scheduler to handle cleaned up containers, which NMs
may subsequently report as running. (Contributed by Vinod Kumar Vavilapalli)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231297
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
Sort benchmark got hang after completion of 99% map phase
-
Key: MAPREDUCE-3596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.23.1
Attachments: MAPREDUCE-3596-20120111.1.txt,
MAPREDUCE-3596-20120111.txt, MAPREDUCE-3596-20120112.1.txt,
MAPREDUCE-3596-20120112.txt, logs.tar.bz2, logs.tar.bz2
Courtesy [~vinaythota]
{quote}
Ran sort benchmark couple of times and every time the job got hang after
completion 99% map phase. There are some map tasks failed. Also it's not
scheduled some of the pending map tasks.
Cluster size is 350 nodes.
Build Details:
==
Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
branches/branch-0.23/hadoop-common-project/hadoop-common
ResourceManager version:revision 1212681 by someone source checksum
on Fri Dec 9 16:52:07 PST 2011
Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
2011
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3656) Sort job on 350 scale is consistently failing with latest MRV2 code
[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186206#comment-13186206
]
Hudson commented on MAPREDUCE-3656:
---
Integrated in Hadoop-Mapreduce-trunk #958 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/958/])
MAPREDUCE-3656. Fixed a race condition in MR AM which is failing the sort
benchmark consistently. Contributed by Siddarth Seth.
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231314
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptListener.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
Sort job on 350 scale is consistently failing with latest MRV2 code
Key: MAPREDUCE-3656
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2, resourcemanager
Affects Versions: 0.23.1
Reporter: Karam Singh
Assignee: Siddharth Seth
Priority: Blocker
Fix For: 0.23.1
Attachments: MR3656.txt, MR3656.txt, MR3656.txt
With the code checked out on last two days.
Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
failing for around last 6 runs
When around 50% of maps are completed, suddenly job jumps to failed state.
On looking at NM log, found RM sent Stop Container Request to NM for AM
container.
But at INFO level from RM log not able find why RM is killing AM when job is
not killed manually.
One thing found common on failed AM logs is -:
org.apache.hadoop.yarn.state.InvalidStateTransitonException
With with different.
For e.g. One log says -:
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
TA_UPDATE at ASSIGNED
{code}
Whereas other logs says -:
{code}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_COUNTER_UPDATE at ERROR
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3404) Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false
[ https://issues.apache.org/jira/browse/MAPREDUCE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186207#comment-13186207 ] Hudson commented on MAPREDUCE-3404: --- Integrated in Hadoop-Mapreduce-trunk #958 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/958/]) MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces. Contributed by Eric Payne. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1231395 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false --- Key: MAPREDUCE-3404 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3404 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, mrv2 Affects Versions: 0.23.0 Environment: Hadoop version is: Hadoop 0.23.0.1110031628 10 node test cluster Reporter: patrick white Assignee: Eric Payne Priority: Critical Fix For: 0.23.1, 0.24.0 Attachments: MAPREDUCE-3404.1.txt, MAPREDUCE-3404.2.txt When forcing a mapper to take significantly longer than other map tasks, speculative map tasks are launched even if the mapreduce.job.maps.speculative.execution parameter is set to 'false'. Testcase: ran default WordCount job with spec execution set to false for both map and reduce but still saw a fifth mapper task launch, ran job as follows: hadoop --config config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=false -Dmapreduce.job.reduces.speculative.execution=false /tmp/test_file_of_words* /tmp/file_of_words.out Input data was 4 text files hdfs blocksize, with same word pattern plus one diff text line in each file, fourth file was 4 times as large as others: hadoop --config config fs -ls /tmp Found 5 items drwxr-xr-x - user hdfs 0 2011-10-20 16:17 /tmp/file_of_words.out -rw-r--r-- 3 user hdfs 62800021 2011-10-20 14:45 /tmp/test_file_of_words1 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words2 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words3 -rw-r--r-- 3 user hdfs 271708312 2011-10-20 15:50 /tmp/test_file_of_words4 Job launched 5 mappers despite spec exec set to false, output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=273540 SLOTS_MILLIS_REDUCES=212876 Reran same case as above only set both spec exec params to 'true', same results only this time the fifth task being launched is expected since spec exec = true. job run: hadoop --config config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=true -Dmapreduce.job.reduces.speculative.execution=true /tmp/test_file_of_words* /tmp/file_of_words.out output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=279653 SLOTS_MILLIS_REDUCES=211474 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3641: - Status: Patch Available (was: Open) Writing unit tests now... CapacityScheduler should be more conservative assigning off-switch requests --- Key: MAPREDUCE-3641 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, scheduler Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, MAPREDUCE-3641.patch In hadoop-1, the CS is very conservative handing out off-switch assignments, we need to do the same in YARN. We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3641: - Attachment: MAPREDUCE-3641.patch Ok, tested patch on secure cluster. CapacityScheduler should be more conservative assigning off-switch requests --- Key: MAPREDUCE-3641 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, scheduler Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, MAPREDUCE-3641.patch In hadoop-1, the CS is very conservative handing out off-switch assignments, we need to do the same in YARN. We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186400#comment-13186400 ] Hadoop QA commented on MAPREDUCE-3641: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510612/MAPREDUCE-3641.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1611//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1611//console This message is automatically generated. CapacityScheduler should be more conservative assigning off-switch requests --- Key: MAPREDUCE-3641 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, scheduler Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, MAPREDUCE-3641.patch In hadoop-1, the CS is very conservative handing out off-switch assignments, we need to do the same in YARN. We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (MAPREDUCE-3673) org.apache.hadoop.mapreduce.lib.chain.ChainMapper missing on 1.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J moved HADOOP-7969 to MAPREDUCE-3673: Component/s: (was: build) mrv1 Target Version/s: 1.1.0 (was: 1.0.0) Affects Version/s: (was: 1.0.0) 1.0.0 Issue Type: Wish (was: New Feature) Key: MAPREDUCE-3673 (was: HADOOP-7969) Project: Hadoop Map/Reduce (was: Hadoop Common) org.apache.hadoop.mapreduce.lib.chain.ChainMapper missing on 1.0 Key: MAPREDUCE-3673 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3673 Project: Hadoop Map/Reduce Issue Type: Wish Components: mrv1 Affects Versions: 1.0.0 Environment: All Reporter: surajz Priority: Minor org.apache.hadoop.mapreduce.lib.chain.ChainMapper is missing on 1.0. if we are using new context api, do we still the ChainMapper/ChainReducer from the mapred package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3673) org.apache.hadoop.mapreduce.lib.chain.ChainMapper missing on 1.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-3673. Resolution: Duplicate 1.0 does not contain the entire set of mapred.lib.* ported to new API. Would you be interested in contributing a backport patch for these classes you demand? Please provide/request on MAPREDUCE-3607 instead, in any case. Marking as duplicate. org.apache.hadoop.mapreduce.lib.chain.ChainMapper missing on 1.0 Key: MAPREDUCE-3673 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3673 Project: Hadoop Map/Reduce Issue Type: Wish Components: mrv1 Affects Versions: 1.0.0 Environment: All Reporter: surajz Priority: Minor org.apache.hadoop.mapreduce.lib.chain.ChainMapper is missing on 1.0. if we are using new context api, do we still the ChainMapper/ChainReducer from the mapred package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
