[jira] [Updated] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-2954:
--
Attachment: MR2954_2.patch
prime back to 31
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2954-20110909.txt, MR2954_1.patch,
> MR2954_2.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101036#comment-13101036
]
Hadoop QA commented on MAPREDUCE-2954:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493735/MAPREDUCE-2954-20110909.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/673//console
This message is automatically generated.
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2954-20110909.txt, MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerM
[jira] [Commented] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101035#comment-13101035
]
Siddharth Seth commented on MAPREDUCE-2954:
---
Looks ok - but am not sure about the large prime - will almost definitely cause
the hashcode to wrap around the integer range which is likely not a problem. We
could revert to the eclipse generated default of 31.
bq. We should be able to do better if we analyse more on our IDs, but this
should work for now.
Completely agree with this though - clusterTimestamp is in ms, there's unlikely
to be a very large number of attemptIds and container per app.
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2954-20110909.txt, MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2965) Streamline hashCode(), equals(), compareTo() and toString() for all IDs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101027#comment-13101027 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2965: Sid or I will take care of this. > Streamline hashCode(), equals(), compareTo() and toString() for all IDs > --- > > Key: MAPREDUCE-2965 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2965 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.0 > > > MAPREDUCE-2954 moved these methods to the record interfaces from the PB impls > for ContainerId, ApplicationId and ApplicationAttemptId. This is good as they > don't need to be tied to the implementation. > We should do the same for all IDs. In fact some of these are missing for IDs > like MR AM JobId, TaskId etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2965) Streamline hashCode(), equals(), compareTo() and toString() for all IDs
Streamline hashCode(), equals(), compareTo() and toString() for all IDs --- Key: MAPREDUCE-2965 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2965 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.0 MAPREDUCE-2954 moved these methods to the record interfaces from the PB impls for ContainerId, ApplicationId and ApplicationAttemptId. This is good as they don't need to be tied to the implementation. We should do the same for all IDs. In fact some of these are missing for IDs like MR AM JobId, TaskId etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2957) The TT should not re-init if it has no good local dirs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101022#comment-13101022 ] Ravi Gummadi commented on MAPREDUCE-2957: - The original implementation in MR-2413 already handles this case of all-disks-failed by throwing Exception from checkLocalDirs(). No ? > The TT should not re-init if it has no good local dirs > -- > > Key: MAPREDUCE-2957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2957 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: tasktracker >Affects Versions: 0.20.204.0 >Reporter: Eli Collins > Fix For: 0.20.205.0 > > > The TT will currently try to re-init itself on disk failure even if it has no > good local dirs. It should shutdown instead. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-2954:
---
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2954-20110909.txt, MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-2954:
---
Attachment: MAPREDUCE-2954-20110909.txt
hashCode() methods generated by eclipse have better null checks etc. and also
have double the product-sums.
Attaching patch using the eclipse-generated hashes. We should be able to do
better if we analyse more on our IDs, but this should work for now.
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2954-20110909.txt, MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2933) Change allocate call to return ContainerStatus for completed containers rather than Container
[ https://issues.apache.org/jira/browse/MAPREDUCE-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2933: - Attachment: MAPREDUCE-2933.patch Rebased. > Change allocate call to return ContainerStatus for completed containers > rather than Container > -- > > Key: MAPREDUCE-2933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2, nodemanager, resourcemanager >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2933.patch, MAPREDUCE-2933.patch, > MAPREDUCE-2933.patch > > > Change allocate call to return ContainerStatus for completed containers > rather than Container, we should do this all the way from the NodeManager too. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2933) Change allocate call to return ContainerStatus for completed containers rather than Container
[ https://issues.apache.org/jira/browse/MAPREDUCE-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2933: - Status: Patch Available (was: Open) > Change allocate call to return ContainerStatus for completed containers > rather than Container > -- > > Key: MAPREDUCE-2933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2933 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2, nodemanager, resourcemanager >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2933.patch, MAPREDUCE-2933.patch, > MAPREDUCE-2933.patch > > > Change allocate call to return ContainerStatus for completed containers > rather than Container, we should do this all the way from the NodeManager too. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-2954:
---
Status: Open (was: Patch Available)
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2964) mapreduce trunk build fails with compile-mapred-test ant target
mapreduce trunk build fails with compile-mapred-test ant target
---
Key: MAPREDUCE-2964
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2964
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: apache jenkins build env:
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/869/console
Reporter: Giridharan Kesavan
{noformat}
compile-mapred-test:
[mkdir] Created dir:
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/mapred/classes
[javac] found : org.apache.hadoop.mapred.IFile.Writer
[javac] required:
org.apache.hadoop.mapred.IFile.Writer
[javac] Writer mockWriter = mock(Writer.class);
[javac] ^
[javac]
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java:128:
warning: [unchecked] unchecked conversion
[javac] found : org.apache.hadoop.mapred.IFile.Writer
[javac] required:
org.apache.hadoop.mapred.IFile.Writer
[javac] Writer mockWriter = mock(Writer.class);
[javac] ^
[javac]
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJvmManager.java:63:
unreported exception java.io.IOException; must be caught or declared to be
thrown
[javac] FileUtil.fullyDelete(TEST_DIR);
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 1 error
[javac] 2 warnings
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:538:
The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:615:
Compile failed; see the compiler error output for details.
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2949) NodeManager in a inconsistent state if a service startup fails.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100997#comment-13100997 ] Arun C Murthy commented on MAPREDUCE-2949: -- Ravi, thanks for taking this up. How does it relate to MAPREDUCE-2956? > NodeManager in a inconsistent state if a service startup fails. > --- > > Key: MAPREDUCE-2949 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2949 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.24.0 >Reporter: Ravi Teja Ch N V >Assignee: Ravi Teja Ch N V > > When a service startup fails at the Nodemanager, the Nodemanager JVM doesnot > exit as the following threads are still running. > Daemon Thread [Timer for 'NodeManager' metrics system] (Running) > Thread [pool-1-thread-1] (Running) > Thread [Thread-11] (Running) > Thread [DestroyJavaVM] (Running). > As a result, the NodeManager keeps running even though no services are > started. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Status: Open (was: Patch Available) Sharad, the approach looks reasonable I'm doing a detailed review. Can you pls rebase against trunk? Thanks. > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal > Attachments: mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool size for container launching in the application master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100996#comment-13100996 ] Arun C Murthy commented on MAPREDUCE-2961: -- Vinod, what heuristic do you have in mind? > Increase the default threadpool size for container launching in the > application master. > --- > > Key: MAPREDUCE-2961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2961.patch > > > Currently the default threadpool size is 10 for launching containers in > ContainerLauncherImpl. We should increase that to 100 for a reasonable > default, so that container launching is not backed up by a small thread pool > size. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100991#comment-13100991
]
Siddharth Seth commented on MAPREDUCE-2954:
---
hashCode() wasn't generated by eclipse. they've taken elements from the MrV1
JobID etc. ApplicationId specifically is something which may need more looking
into (post RM re-factor and for JobHistory).
Agree with the bit about tying equals(), hashCode, compareTo.. and also
toString() to all the records. ProtoBase is more of a convenience to provide
this functionality in all records - and likely the reason for most of the races
and sync in PBImpls (serializing the proto object for each equals, hashCode).
This - along with some other PB performance related changes need to be made
sometime later.
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-2711:
--
Resolution: Fixed
Fix Version/s: 0.24.0
0.23.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Thanks Arun for the review.
I have committed this.
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.23.0, 0.24.0
>
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
> {{TestBlockPlacementPolicyRaid}} access internal {{FSNamesystem}} directly.
> It cannot be compiled after HDFS-2147.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100986#comment-13100986
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2954:
Patch looks good overall.
Assuming the {{hashCode()}} methods are generated(eclipse?) and good enough.
I also like the fact that now _equals()_, _hashCode()_, and _comparesTo()_ are
tied to the records themselves instead of the PB implementations. We should do
this for other records too, but another ticket.
+1 for the patch.
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100980#comment-13100980
]
Hudson commented on MAPREDUCE-2711:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #869 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/869/])
MAPREDUCE-2711. Update TestBlockPlacementPolicyRaid for the new namesystem
and block management APIs.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1167001
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidTestUtil.java
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
> {{TestBlockPlacementPolicyRaid}} access internal {{FSNamesystem}} directly.
> It cannot be compiled after HDFS-2147.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2656) Map Reduce Tasks are continously failing, when one among the several harddisks available on the TaskTracker fails.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100979#comment-13100979
]
Devaraj K commented on MAPREDUCE-2656:
--
This is not related to MAPREDUCE-2647. That feature can be provided as
contrib/tool and whoever interested can use it. We can have further discussions
in that issue.
This issue exists only in 0.20.2 version hence resolving it.
> Map Reduce Tasks are continously failing, when one among the several
> harddisks available on the TaskTracker fails.
> --
>
> Key: MAPREDUCE-2656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2656
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Affects Versions: 0.20.2, 0.20.3
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.20.2, 0.20.3
>
> Attachments: HADOOP-7130.patch, MAPREDUCE-2656.patch
>
>
> 1. Pull out one hard disk from Task tracker node (out of 10 disks pull one).
> Now it is noted that some jobs are failing.
> However process is continued.
> 2. Wait for sometime (15 mins) and pull out one disk from another Task
> tracker.
> 3. More number of jobs failed now and it can be seen from UI. Process is
> getting paused.
> The exception can be seen in the job tracker UI for a failed job.
> {code:xml}
> Error initializing attempt_201010221528_10174_m_11_0:
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:385)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
> at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
> at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> Error initializing attempt_201010221528_10174_m_11_1:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for taskTracker/jobcache/job_201010221528_10174/work
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:454)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
> at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
> at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> {code}
> Task Tracker log can be seen here :
> {code:xml}
> 2010-10-25 16:36:24,215 ERROR mapred.TaskTracker
> (TaskTracker.java:offerService(1211)) - Caught exception:
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
> at
> org.apache.hadoop.mapred.TaskTracker.getFreeSpace(TaskTracker.java:1586)
> at
> org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1274)
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1106)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> 2010-10-25 16:36:24,216 INFO mapred.TaskTracker (TaskTracker.java:run(1856))
> - Lost connection to JobTracker [/192.168.97.1:9001]. Retrying...
> java.lang.Exception: java.io.IOException: Expecting a line not the end of
> stream
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1212)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> Caused by: java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> a
[jira] [Updated] (MAPREDUCE-2656) Map Reduce Tasks are continously failing, when one among the several harddisks available on the TaskTracker fails.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-2656:
-
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
> Map Reduce Tasks are continously failing, when one among the several
> harddisks available on the TaskTracker fails.
> --
>
> Key: MAPREDUCE-2656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2656
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Affects Versions: 0.20.2, 0.20.3
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.20.2, 0.20.3
>
> Attachments: HADOOP-7130.patch, MAPREDUCE-2656.patch
>
>
> 1. Pull out one hard disk from Task tracker node (out of 10 disks pull one).
> Now it is noted that some jobs are failing.
> However process is continued.
> 2. Wait for sometime (15 mins) and pull out one disk from another Task
> tracker.
> 3. More number of jobs failed now and it can be seen from UI. Process is
> getting paused.
> The exception can be seen in the job tracker UI for a failed job.
> {code:xml}
> Error initializing attempt_201010221528_10174_m_11_0:
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:385)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
> at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
> at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> Error initializing attempt_201010221528_10174_m_11_1:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for taskTracker/jobcache/job_201010221528_10174/work
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:454)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
> at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
> at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> {code}
> Task Tracker log can be seen here :
> {code:xml}
> 2010-10-25 16:36:24,215 ERROR mapred.TaskTracker
> (TaskTracker.java:offerService(1211)) - Caught exception:
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
> at
> org.apache.hadoop.mapred.TaskTracker.getFreeSpace(TaskTracker.java:1586)
> at
> org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1274)
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1106)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> 2010-10-25 16:36:24,216 INFO mapred.TaskTracker (TaskTracker.java:run(1856))
> - Lost connection to JobTracker [/192.168.97.1:9001]. Retrying...
> java.lang.Exception: java.io.IOException: Expecting a line not the end of
> stream
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1212)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> Caused by: java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
> at
> org.apache.hadoop.mapred.TaskTracker.getFreeSpace(TaskTracker.java:1586)
> at
[jira] [Commented] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100977#comment-13100977
]
Hudson commented on MAPREDUCE-2711:
---
Integrated in Hadoop-Hdfs-trunk-Commit #935 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/935/])
MAPREDUCE-2711. Update TestBlockPlacementPolicyRaid for the new namesystem
and block management APIs.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1167001
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidTestUtil.java
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
> {{TestBlockPlacementPolicyRaid}} access internal {{FSNamesystem}} directly.
> It cannot be compiled after HDFS-2147.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100975#comment-13100975
]
Hudson commented on MAPREDUCE-2711:
---
Integrated in Hadoop-Common-trunk-Commit #858 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/858/])
MAPREDUCE-2711. Update TestBlockPlacementPolicyRaid for the new namesystem
and block management APIs.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1167001
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockPlacementPolicyRaid.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/raid/src/test/org/apache/hadoop/hdfs/server/namenode/NameNodeRaidTestUtil.java
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
> {{TestBlockPlacementPolicyRaid}} access internal {{FSNamesystem}} directly.
> It cannot be compiled after HDFS-2147.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2949) NodeManager in a inconsistent state if a service startup fails.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Teja Ch N V reassigned MAPREDUCE-2949: --- Assignee: Ravi Teja Ch N V > NodeManager in a inconsistent state if a service startup fails. > --- > > Key: MAPREDUCE-2949 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2949 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.24.0 >Reporter: Ravi Teja Ch N V >Assignee: Ravi Teja Ch N V > > When a service startup fails at the Nodemanager, the Nodemanager JVM doesnot > exit as the following threads are still running. > Daemon Thread [Timer for 'NodeManager' metrics system] (Running) > Thread [pool-1-thread-1] (Running) > Thread [Thread-11] (Running) > Thread [DestroyJavaVM] (Running). > As a result, the NodeManager keeps running even though no services are > started. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2656) Map Reduce Tasks are continously failing, when one among the several harddisks available on the TaskTracker fails.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100961#comment-13100961
]
Arun C Murthy commented on MAPREDUCE-2656:
--
Sorry, I meant to MAPREDUCE-2647 vis-a-vis 0.20.205.
My proposal is we drop this for 0.20.3 which is unlikely to be released now,
afaics.
> Map Reduce Tasks are continously failing, when one among the several
> harddisks available on the TaskTracker fails.
> --
>
> Key: MAPREDUCE-2656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2656
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Affects Versions: 0.20.2, 0.20.3
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.20.2, 0.20.3
>
> Attachments: HADOOP-7130.patch, MAPREDUCE-2656.patch
>
>
> 1. Pull out one hard disk from Task tracker node (out of 10 disks pull one).
> Now it is noted that some jobs are failing.
> However process is continued.
> 2. Wait for sometime (15 mins) and pull out one disk from another Task
> tracker.
> 3. More number of jobs failed now and it can be seen from UI. Process is
> getting paused.
> The exception can be seen in the job tracker UI for a failed job.
> {code:xml}
> Error initializing attempt_201010221528_10174_m_11_0:
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:385)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
> at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
> at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> Error initializing attempt_201010221528_10174_m_11_1:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for taskTracker/jobcache/job_201010221528_10174/work
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:454)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
> at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
> at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> {code}
> Task Tracker log can be seen here :
> {code:xml}
> 2010-10-25 16:36:24,215 ERROR mapred.TaskTracker
> (TaskTracker.java:offerService(1211)) - Caught exception:
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
> at
> org.apache.hadoop.mapred.TaskTracker.getFreeSpace(TaskTracker.java:1586)
> at
> org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1274)
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1106)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> 2010-10-25 16:36:24,216 INFO mapred.TaskTracker (TaskTracker.java:run(1856))
> - Lost connection to JobTracker [/192.168.97.1:9001]. Retrying...
> java.lang.Exception: java.io.IOException: Expecting a line not the end of
> stream
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1212)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> Caused by: java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool size for container launching in the application master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100963#comment-13100963 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2961: I definitely think that instead of a config, this should be a function of #nodes and #tasks with defaults to 10. Will provide a patch for the same. > Increase the default threadpool size for container launching in the > application master. > --- > > Key: MAPREDUCE-2961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2961.patch > > > Currently the default threadpool size is 10 for launching containers in > ContainerLauncherImpl. We should increase that to 100 for a reasonable > default, so that container launching is not backed up by a small thread pool > size. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2656) Map Reduce Tasks are continously failing, when one among the several harddisks available on the TaskTracker fails.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100959#comment-13100959
]
Devaraj K commented on MAPREDUCE-2656:
--
@Eli : This issue is already rectified in branch-20-security. This patch is
relevant to 20.2. If it is not useful further, we can close this issue.
@Arun: It doesn't relate to MAPREDUCE-2143. I am not getting why MAPREDUCE-2143
has come into picture.
> Map Reduce Tasks are continously failing, when one among the several
> harddisks available on the TaskTracker fails.
> --
>
> Key: MAPREDUCE-2656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2656
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Affects Versions: 0.20.2, 0.20.3
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 0.20.2, 0.20.3
>
> Attachments: HADOOP-7130.patch, MAPREDUCE-2656.patch
>
>
> 1. Pull out one hard disk from Task tracker node (out of 10 disks pull one).
> Now it is noted that some jobs are failing.
> However process is continued.
> 2. Wait for sometime (15 mins) and pull out one disk from another Task
> tracker.
> 3. More number of jobs failed now and it can be seen from UI. Process is
> getting paused.
> The exception can be seen in the job tracker UI for a failed job.
> {code:xml}
> Error initializing attempt_201010221528_10174_m_11_0:
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:385)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
> at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
> at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> Error initializing attempt_201010221528_10174_m_11_1:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
> valid local directory for taskTracker/jobcache/job_201010221528_10174/work
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:454)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
> at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
> at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
> at
> org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> {code}
> Task Tracker log can be seen here :
> {code:xml}
> 2010-10-25 16:36:24,215 ERROR mapred.TaskTracker
> (TaskTracker.java:offerService(1211)) - Caught exception:
> java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
> at org.apache.hadoop.util.Shell.run(Shell.java:137)
> at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
> at
> org.apache.hadoop.mapred.TaskTracker.getFreeSpace(TaskTracker.java:1586)
> at
> org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1274)
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1106)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> 2010-10-25 16:36:24,216 INFO mapred.TaskTracker (TaskTracker.java:run(1856))
> - Lost connection to JobTracker [/192.168.97.1:9001]. Retrying...
> java.lang.Exception: java.io.IOException: Expecting a line not the end of
> stream
> at
> org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1212)
> at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
> at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> Caused by: java.io.IOException: Expecting a line not the end of stream
> at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
> at org.apache.hadoop.util.Shell.runCommand(Shell
[jira] [Commented] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100955#comment-13100955
]
Arun C Murthy commented on MAPREDUCE-2711:
--
+1
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
> {{TestBlockPlacementPolicyRaid}} access internal {{FSNamesystem}} directly.
> It cannot be compiled after HDFS-2147.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2952) Application failure diagnostics are not consumed in a couple of cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100952#comment-13100952 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2952: bq. Vinod, you mean MAPREDUCE-2937. Yes. > Application failure diagnostics are not consumed in a couple of cases > - > > Key: MAPREDUCE-2952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 0.23.0 > > > When Container crashes, the reason for failures isn't propagated because of a > bug in _RMAppAttemptImpl.AMContainerCrashedTransition_ which simply discards > the diagnostics of the container. Also RMAppAttemptImpl.diagnostics is never > consumed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100950#comment-13100950
]
Hadoop QA commented on MAPREDUCE-2711:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493722/m2711_20110908.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/672//console
This message is automatically generated.
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
> {{TestBlockPlacementPolicyRaid}} access internal {{FSNamesystem}} directly.
> It cannot be compiled after HDFS-2147.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100949#comment-13100949
]
Hadoop QA commented on MAPREDUCE-2711:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493722/m2711_20110908.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/671//console
This message is automatically generated.
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
> {{TestBlockPlacementPolicyRaid}} access internal {{FSNamesystem}} directly.
> It cannot be compiled after HDFS-2147.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2691) Finish up the cleanup of distributed cache file resources and related tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100945#comment-13100945 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-2691: bq. Draining events immediately - don't quite remember why I added the option for a delayed drain The delayed drain was already there on trunk, just not used. bq. Yep, deletion exists in MRv2 and works. For LRU, ref MR 2494, 2572. Thanks, I'll file corresponding ticket for MRV2. > Finish up the cleanup of distributed cache file resources and related tests. > > > Key: MAPREDUCE-2691 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2691 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Amol Kekre >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2691_1.patch, MR2691_2.patch, MR2691_3.patch > > > Implement cleanup of distributed cache file resources -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-2711:
--
Description: {{TestBlockPlacementPolicyRaid}} access internal
{{FSNamesystem}} directly. It cannot be compiled after HDFS-2147.
Environment: (was: {{TestBlockPlacementPolicyRaid}} access internal
{{FSNamesystem}} directly. It cannot be compiled after HDFS-2147.)
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
> {{TestBlockPlacementPolicyRaid}} access internal {{FSNamesystem}} directly.
> It cannot be compiled after HDFS-2147.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-2711:
--
Status: Patch Available (was: In Progress)
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
> Environment: {{TestBlockPlacementPolicyRaid}} access internal
> {{FSNamesystem}} directly. It cannot be compiled after HDFS-2147.
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-2711:
--
Attachment: m2711_20110908.patch
m2711_20110908.patch: updated with trunk
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
> Environment: {{TestBlockPlacementPolicyRaid}} access internal
> {{FSNamesystem}} directly. It cannot be compiled after HDFS-2147.
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch, m2711_20110908.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2952) Application failure diagnostics are not consumed in a couple of cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2952: - Priority: Blocker (was: Major) > Application failure diagnostics are not consumed in a couple of cases > - > > Key: MAPREDUCE-2952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 0.23.0 > > > When Container crashes, the reason for failures isn't propagated because of a > bug in _RMAppAttemptImpl.AMContainerCrashedTransition_ which simply discards > the diagnostics of the container. Also RMAppAttemptImpl.diagnostics is never > consumed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2675) MR-279: JobHistory Server main page needs to be reformatted
[ https://issues.apache.org/jira/browse/MAPREDUCE-2675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2675: - Status: Open (was: Patch Available) Cancelling patch while Sid's feedback is incorporated. > MR-279: JobHistory Server main page needs to be reformatted > --- > > Key: MAPREDUCE-2675 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2675 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2675-V1.patch, MR-2675-V2.patch, MR-2675-V3.patch, > MR-2675-V4.patch, MR-2675-preview.patch > > > The main page of the Job History Server is based off of the Application > Master code. It needs to be reformatted to be more useful and better match > what was there before. > - The Active Jobs title needs to be replaced with something more appropriate > (i.e. Retired Jobs) > - The table of jobs should have the following columns in it > - Submit time, Job Id, Job Name, User and just because I think it would be > useful state, maps completed, maps failed, reduces completed, reduces failed > - The table needs more advanced filtering, something like > http://datatables.net/release-datatables/examples/api/multi_filter.html This > is to match the previous search functionality. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool size for container launching in the application master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2961: - Status: Open (was: Patch Available) Mahadev, you need to rebase after MAPREDUCE-2864. Thanks. > Increase the default threadpool size for container launching in the > application master. > --- > > Key: MAPREDUCE-2961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2961.patch > > > Currently the default threadpool size is 10 for launching containers in > ContainerLauncherImpl. We should increase that to 100 for a reasonable > default, so that container launching is not backed up by a small thread pool > size. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2691) Finish up the cleanup of distributed cache file resources and related tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100916#comment-13100916 ] Siddharth Seth commented on MAPREDUCE-2691: --- Thanks for the detailed review Vinod. ContainerImpl.handle and ContainerManager.stopContainer()diagnostic - will revert the changes. Vaguely remember seing diagnostic messages stacking up - where a single container would end up with multiple copies of messages. Will create a jira for that when I see it again. bq. RELEASE_CONTAINER_RESOURCES is always sent along with CLEANUP_CONTAINER_RESOURCES event. I think we should just merge these into CLEANUP_CONTAINER_RESOURCES event itself. This will also be inline with the fact that creation of container-dirs and the localization of files both happen as part of a single event INIT_CONTAINER_RESOURCES, so cleanup should also be a single event. We can send a Map> as the event payload. To be symmetric, we should probably also merge the multiple INIT_CONTAINER_RESOURCES calls one for each LocalResourceVisibility to be a single event. Thoughts? Sounds good. Will make the changes. Had added separate events for RELEASE_CONTAINER_RESOURCES to be consistent with the way resources were requested - 1 event for each type. Don't really see a reason for the requests to be sent separately though. Will make the changes in the test cases. Draining events immediately - don't quite remember why I added the option for a delayed drain - possibly to be able to drain events 1 at a time sometime later. Anyway, it can be added back if required. Completely agree about the mocks - there's way too much and makes some of the tests hard to understand. Will try getting rid of some of them. bq. There is existing code for purging of cache under disk pressure - See ResourceLocalization.CacheCleanup and ResourceRetentionSet. (We need tests for this though, will file a ticket) This only deletes files that aren't in use at all. By LRU, do you mean selective deletion of these files based on their usage? Can you please point me to the relevant MRV1 JIRA? Thanks! Yep, deletion exists in MRv2 and works. For LRU, ref MR 2494, 2572. > Finish up the cleanup of distributed cache file resources and related tests. > > > Key: MAPREDUCE-2691 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2691 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Amol Kekre >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2691_1.patch, MR2691_2.patch, MR2691_3.patch > > > Implement cleanup of distributed cache file resources -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100907#comment-13100907 ] Hudson commented on MAPREDUCE-2963: --- Integrated in Hadoop-Mapreduce-trunk-Commit #868 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/868/]) MAPREDUCE-2963. Fix hang in TestMRJobs. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166975 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java > TestMRJobs hangs waiting to connect to history server. > -- > > Key: MAPREDUCE-2963 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.24.0 >Reporter: Mahadev konar >Assignee: Siddharth Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR2963_v1.patch > > > TestMRJobs is hanging waiting to connect to history server. I will post the > logs next. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100906#comment-13100906 ] Hudson commented on MAPREDUCE-279: -- Integrated in Hadoop-Mapreduce-trunk-Commit #868 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/868/]) Adding back hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources which was missed during the merge of MAPREDUCE-279. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166972 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider > Map-Reduce 2.0 > -- > > Key: MAPREDUCE-279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Arun C Murthy > Fix For: 0.23.0 > > Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, > MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, > MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, > MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, > NodeManager.gv, NodeManager.png, ResourceManager.gv, ResourceManager.png, > capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, > multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, > post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, > yarn-state-machine.job.dot, yarn-state-machine.job.png, > yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, > yarn-state-machine.task.dot, yarn-state-machine.task.png > > > Re-factor MapReduce into a generic resource scheduler and a per-job, > user-defined component that manages the application execution. > Check it out by following [the instructions|http://goo.gl/rSJJC]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100905#comment-13100905 ] Hudson commented on MAPREDUCE-2963: --- Integrated in Hadoop-Hdfs-trunk-Commit #934 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/934/]) MAPREDUCE-2963. Fix hang in TestMRJobs. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166975 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java > TestMRJobs hangs waiting to connect to history server. > -- > > Key: MAPREDUCE-2963 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.24.0 >Reporter: Mahadev konar >Assignee: Siddharth Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR2963_v1.patch > > > TestMRJobs is hanging waiting to connect to history server. I will post the > logs next. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100904#comment-13100904 ] Hudson commented on MAPREDUCE-279: -- Integrated in Hadoop-Hdfs-trunk-Commit #934 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/934/]) Adding back hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources which was missed during the merge of MAPREDUCE-279. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166972 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider > Map-Reduce 2.0 > -- > > Key: MAPREDUCE-279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Arun C Murthy > Fix For: 0.23.0 > > Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, > MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, > MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, > MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, > NodeManager.gv, NodeManager.png, ResourceManager.gv, ResourceManager.png, > capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, > multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, > post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, > yarn-state-machine.job.dot, yarn-state-machine.job.png, > yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, > yarn-state-machine.task.dot, yarn-state-machine.task.png > > > Re-factor MapReduce into a generic resource scheduler and a per-job, > user-defined component that manages the application execution. > Check it out by following [the instructions|http://goo.gl/rSJJC]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-279) Map-Reduce 2.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100902#comment-13100902 ] Hudson commented on MAPREDUCE-279: -- Integrated in Hadoop-Common-trunk-Commit #857 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/857/]) Adding back hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources which was missed during the merge of MAPREDUCE-279. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166972 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.mapreduce.protocol.ClientProtocolProvider > Map-Reduce 2.0 > -- > > Key: MAPREDUCE-279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-279 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Arun C Murthy > Fix For: 0.23.0 > > Attachments: MR-279-script-20110817.sh, MR-279-script-final.sh, > MR-279-script.sh, MR-279-script.sh, MR-279.patch, MR-279.patch, MR-279.sh, > MR-279_MR_files_to_move-20110817.txt, MR-279_MR_files_to_move.txt, > MR-279_MR_files_to_move.txt, MapReduce_NextGen_Architecture.pdf, > NodeManager.gv, NodeManager.png, ResourceManager.gv, ResourceManager.png, > capacity-scheduler-dark-theme.png, hadoop_contributors_meet_07_01_2011.pdf, > multi-column-stable-sort-default-theme.png, post-move-patch-20110817.2.txt, > post-move-patch-final.txt, post-move.patch, post-move.patch, post-move.patch, > yarn-state-machine.job.dot, yarn-state-machine.job.png, > yarn-state-machine.task-attempt.dot, yarn-state-machine.task-attempt.png, > yarn-state-machine.task.dot, yarn-state-machine.task.png > > > Re-factor MapReduce into a generic resource scheduler and a per-job, > user-defined component that manages the application execution. > Check it out by following [the instructions|http://goo.gl/rSJJC]. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100903#comment-13100903 ] Hudson commented on MAPREDUCE-2963: --- Integrated in Hadoop-Common-trunk-Commit #857 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/857/]) MAPREDUCE-2963. Fix hang in TestMRJobs. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166975 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java > TestMRJobs hangs waiting to connect to history server. > -- > > Key: MAPREDUCE-2963 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.24.0 >Reporter: Mahadev konar >Assignee: Siddharth Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR2963_v1.patch > > > TestMRJobs is hanging waiting to connect to history server. I will post the > logs next. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2963: - Resolution: Fixed Fix Version/s: (was: 0.24.0) Status: Resolved (was: Patch Available) I just committed this. Thanks Sid! > TestMRJobs hangs waiting to connect to history server. > -- > > Key: MAPREDUCE-2963 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.24.0 >Reporter: Mahadev konar >Assignee: Siddharth Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR2963_v1.patch > > > TestMRJobs is hanging waiting to connect to history server. I will post the > logs next. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100899#comment-13100899 ] Hadoop QA commented on MAPREDUCE-2963: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493719/MR2963_v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/670//console This message is automatically generated. > TestMRJobs hangs waiting to connect to history server. > -- > > Key: MAPREDUCE-2963 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.24.0 >Reporter: Mahadev konar >Assignee: Siddharth Seth >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2963_v1.patch > > > TestMRJobs is hanging waiting to connect to history server. I will post the > logs next. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-2711) TestBlockPlacementPolicyRaid cannot be compiled
[
https://issues.apache.org/jira/browse/MAPREDUCE-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on MAPREDUCE-2711 started by Tsz Wo (Nicholas), SZE.
> TestBlockPlacementPolicyRaid cannot be compiled
> ---
>
> Key: MAPREDUCE-2711
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2711
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
> Environment: {{TestBlockPlacementPolicyRaid}} access internal
> {{FSNamesystem}} directly. It cannot be compiled after HDFS-2147.
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: m2711_20110719_TestBlockPlacementPolicyRaid.java,
> m2711_20110727.patch, m2711_20110818.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2952) Application failure diagnostics are not consumed in a couple of cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100896#comment-13100896 ] Arun C Murthy commented on MAPREDUCE-2952: -- Vinod, you mean MAPREDUCE-2937. > Application failure diagnostics are not consumed in a couple of cases > - > > Key: MAPREDUCE-2952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli > Fix For: 0.23.0 > > > When Container crashes, the reason for failures isn't propagated because of a > bug in _RMAppAttemptImpl.AMContainerCrashedTransition_ which simply discards > the diagnostics of the container. Also RMAppAttemptImpl.diagnostics is never > consumed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2958) mapred-default.xml not merged from mr279
[ https://issues.apache.org/jira/browse/MAPREDUCE-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-2958. -- Resolution: Fixed Assignee: Arun C Murthy (was: Mahadev konar) Thanks Thomas, I fixed it by doing an 'svn copy' to both trunk and branch-0.23. > mapred-default.xml not merged from mr279 > > > Key: MAPREDUCE-2958 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2958 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Arun C Murthy >Priority: Critical > Fix For: 0.23.0 > > > I have been running wordcount out of the 23 examples jar. It says it > succeeds but doesn't actually output a file. > hadoop jar > examples/hadoop-mapreduce-0.23.0-SNAPSHOT/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar > wordcount input output2 > input file is really basic: > fdksajl > dlkfsajlfljda;j > kldfsjallj > test > one > two > test -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2963: -- Attachment: MR2963_v1.patch Trivial patch - getTaskReports wasn't setting request params. Haven't included any tests. > TestMRJobs hangs waiting to connect to history server. > -- > > Key: MAPREDUCE-2963 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.24.0 >Reporter: Mahadev konar >Assignee: Siddharth Seth >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2963_v1.patch > > > TestMRJobs is hanging waiting to connect to history server. I will post the > logs next. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-2963: -- Fix Version/s: 0.24.0 Affects Version/s: 0.24.0 Status: Patch Available (was: Open) > TestMRJobs hangs waiting to connect to history server. > -- > > Key: MAPREDUCE-2963 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0, 0.24.0 >Reporter: Mahadev konar >Assignee: Siddharth Seth >Priority: Critical > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2963_v1.patch > > > TestMRJobs is hanging waiting to connect to history server. I will post the > logs next. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reassigned MAPREDUCE-2963: - Assignee: Siddharth Seth > TestMRJobs hangs waiting to connect to history server. > -- > > Key: MAPREDUCE-2963 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Siddharth Seth >Priority: Critical > Fix For: 0.23.0 > > > TestMRJobs is hanging waiting to connect to history server. I will post the > logs next. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2958) mapred-default.xml not merged from mr279
[ https://issues.apache.org/jira/browse/MAPREDUCE-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2958: - Assignee: Mahadev konar (was: Thomas Graves) Summary: mapred-default.xml not merged from mr279 (was: wordcount succeeds but doesn't output file) > mapred-default.xml not merged from mr279 > > > Key: MAPREDUCE-2958 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2958 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Mahadev konar >Priority: Critical > Fix For: 0.23.0 > > > I have been running wordcount out of the 23 examples jar. It says it > succeeds but doesn't actually output a file. > hadoop jar > examples/hadoop-mapreduce-0.23.0-SNAPSHOT/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar > wordcount input output2 > input file is really basic: > fdksajl > dlkfsajlfljda;j > kldfsjallj > test > one > two > test -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2958) wordcount succeeds but doesn't output file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100889#comment-13100889 ] Thomas Graves commented on MAPREDUCE-2958: -- it looks like the ./mr-client/hadoop-mapreduce-client-core/src/main/resources/ directory never got merged to trunk from mr-279 branch. That directory included the mapred-default.xml that included the default reducers. Assigned to Mahadev to investigate if it was intentional or accidentally missed during the merge. > wordcount succeeds but doesn't output file > -- > > Key: MAPREDUCE-2958 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2958 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.0 > > > I have been running wordcount out of the 23 examples jar. It says it > succeeds but doesn't actually output a file. > hadoop jar > examples/hadoop-mapreduce-0.23.0-SNAPSHOT/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar > wordcount input output2 > input file is really basic: > fdksajl > dlkfsajlfljda;j > kldfsjallj > test > one > two > test -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2953) JobClient fails due to a race in RM, removes staged files and in turn crashes MR AM
[
https://issues.apache.org/jira/browse/MAPREDUCE-2953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100883#comment-13100883
]
Hudson commented on MAPREDUCE-2953:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #867 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/867/])
MAPREDUCE-2953. Fix a race condition on submission which caused client to
incorrectly assume application was gone by making submission synchronous for
RMAppManager. Contributed by Thomas Graves.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166968
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
> JobClient fails due to a race in RM, removes staged files and in turn crashes
> MR AM
> ---
>
> Key: MAPREDUCE-2953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2953
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2953-v2.patch, MAPREDUCE-2953-v3.patch,
> MAPREDUCE-2953.patch
>
>
> [~Karams] ran into this multiple times. MR JobClient crashes immediately.
> {code}
> 11/09/08 10:52:35 INFO mapreduce.JobSubmitter: number of splits:2094
> 11/09/08 10:52:36 INFO mapred.YARNRunner: AppMaster capability = memory: 2048,
> 11/09/08 10:52:36 INFO mapred.YARNRunner: Command to launch container for
> ApplicationMaster is : $JAVA_HOME/bin/java -Dhadoop.root.logger=INFO,console
> -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1315478927026 1
> 1>/stdout 2>/stderr
> 11/09/08 10:52:36 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1315478927026_1 to ResourceManager
> 11/09/08 10:52:36 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user/gridperf/.staging/job_1315478927026_0001
> RemoteTrace:
> at Local Trace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> failed to run job
> at
> org.apache.hadoop.yarn.factories.impl.pb.YarnRemoteExceptionFactoryPBImpl.createYarnRemoteException(YarnRemoteExceptionFactoryPBImpl.java:39)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:47)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:250)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:377)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:283)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:294)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.D
[jira] [Commented] (MAPREDUCE-2937) Errors in Application failures are not shown in the client trace.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100884#comment-13100884 ] Hudson commented on MAPREDUCE-2937: --- Integrated in Hadoop-Mapreduce-trunk-Commit #867 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/867/]) MAPREDUCE-2937. Ensure reason for application failure is displayed to the user. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166966 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestYARNRunner.java > Errors in Application failures are not shown in the client trace. > - > > Key: MAPREDUCE-2937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2937 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2937.patch, MAPREDUCE-2937.patch, > MAPREDUCE-2937.patch > > > The client side does not show enough information on why the job failed. Here > is step to reproduce it: > 1) set the scheduler to be capacity scheduler with queues a, b > 2) submit a job to a queue that is not a,b > The job just fails without saying why it failed. We should have enough trace > log at the client side to let the user know why it failed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2953) JobClient fails due to a race in RM, removes staged files and in turn crashes MR AM
[
https://issues.apache.org/jira/browse/MAPREDUCE-2953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100879#comment-13100879
]
Hudson commented on MAPREDUCE-2953:
---
Integrated in Hadoop-Hdfs-trunk-Commit #933 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/933/])
MAPREDUCE-2953. Fix a race condition on submission which caused client to
incorrectly assume application was gone by making submission synchronous for
RMAppManager. Contributed by Thomas Graves.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166968
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
> JobClient fails due to a race in RM, removes staged files and in turn crashes
> MR AM
> ---
>
> Key: MAPREDUCE-2953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2953
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2953-v2.patch, MAPREDUCE-2953-v3.patch,
> MAPREDUCE-2953.patch
>
>
> [~Karams] ran into this multiple times. MR JobClient crashes immediately.
> {code}
> 11/09/08 10:52:35 INFO mapreduce.JobSubmitter: number of splits:2094
> 11/09/08 10:52:36 INFO mapred.YARNRunner: AppMaster capability = memory: 2048,
> 11/09/08 10:52:36 INFO mapred.YARNRunner: Command to launch container for
> ApplicationMaster is : $JAVA_HOME/bin/java -Dhadoop.root.logger=INFO,console
> -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1315478927026 1
> 1>/stdout 2>/stderr
> 11/09/08 10:52:36 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1315478927026_1 to ResourceManager
> 11/09/08 10:52:36 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user/gridperf/.staging/job_1315478927026_0001
> RemoteTrace:
> at Local Trace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> failed to run job
> at
> org.apache.hadoop.yarn.factories.impl.pb.YarnRemoteExceptionFactoryPBImpl.createYarnRemoteException(YarnRemoteExceptionFactoryPBImpl.java:39)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:47)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:250)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:377)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:283)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:294)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingM
[jira] [Commented] (MAPREDUCE-2937) Errors in Application failures are not shown in the client trace.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100880#comment-13100880 ] Hudson commented on MAPREDUCE-2937: --- Integrated in Hadoop-Hdfs-trunk-Commit #933 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/933/]) MAPREDUCE-2937. Ensure reason for application failure is displayed to the user. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166966 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestYARNRunner.java > Errors in Application failures are not shown in the client trace. > - > > Key: MAPREDUCE-2937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2937 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2937.patch, MAPREDUCE-2937.patch, > MAPREDUCE-2937.patch > > > The client side does not show enough information on why the job failed. Here > is step to reproduce it: > 1) set the scheduler to be capacity scheduler with queues a, b > 2) submit a job to a queue that is not a,b > The job just fails without saying why it failed. We should have enough trace > log at the client side to let the user know why it failed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2953) JobClient fails due to a race in RM, removes staged files and in turn crashes MR AM
[
https://issues.apache.org/jira/browse/MAPREDUCE-2953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100876#comment-13100876
]
Hudson commented on MAPREDUCE-2953:
---
Integrated in Hadoop-Common-trunk-Commit #856 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/856/])
MAPREDUCE-2953. Fix a race condition on submission which caused client to
incorrectly assume application was gone by making submission synchronous for
RMAppManager. Contributed by Thomas Graves.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166968
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
> JobClient fails due to a race in RM, removes staged files and in turn crashes
> MR AM
> ---
>
> Key: MAPREDUCE-2953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2953
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2953-v2.patch, MAPREDUCE-2953-v3.patch,
> MAPREDUCE-2953.patch
>
>
> [~Karams] ran into this multiple times. MR JobClient crashes immediately.
> {code}
> 11/09/08 10:52:35 INFO mapreduce.JobSubmitter: number of splits:2094
> 11/09/08 10:52:36 INFO mapred.YARNRunner: AppMaster capability = memory: 2048,
> 11/09/08 10:52:36 INFO mapred.YARNRunner: Command to launch container for
> ApplicationMaster is : $JAVA_HOME/bin/java -Dhadoop.root.logger=INFO,console
> -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1315478927026 1
> 1>/stdout 2>/stderr
> 11/09/08 10:52:36 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1315478927026_1 to ResourceManager
> 11/09/08 10:52:36 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user/gridperf/.staging/job_1315478927026_0001
> RemoteTrace:
> at Local Trace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> failed to run job
> at
> org.apache.hadoop.yarn.factories.impl.pb.YarnRemoteExceptionFactoryPBImpl.createYarnRemoteException(YarnRemoteExceptionFactoryPBImpl.java:39)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:47)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:250)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:377)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:283)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:294)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.Delegat
[jira] [Commented] (MAPREDUCE-2937) Errors in Application failures are not shown in the client trace.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100877#comment-13100877 ] Hudson commented on MAPREDUCE-2937: --- Integrated in Hadoop-Common-trunk-Commit #856 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/856/]) MAPREDUCE-2937. Ensure reason for application failure is displayed to the user. Contributed by Mahadev Konar. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166966 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestYARNRunner.java > Errors in Application failures are not shown in the client trace. > - > > Key: MAPREDUCE-2937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2937 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2937.patch, MAPREDUCE-2937.patch, > MAPREDUCE-2937.patch > > > The client side does not show enough information on why the job failed. Here > is step to reproduce it: > 1) set the scheduler to be capacity scheduler with queues a, b > 2) submit a job to a queue that is not a,b > The job just fails without saying why it failed. We should have enough trace > log at the client side to let the user know why it failed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2953) JobClient fails due to a race in RM, removes staged files and in turn crashes MR AM
[
https://issues.apache.org/jira/browse/MAPREDUCE-2953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2953:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Thomas!
> JobClient fails due to a race in RM, removes staged files and in turn crashes
> MR AM
> ---
>
> Key: MAPREDUCE-2953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2953
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2953-v2.patch, MAPREDUCE-2953-v3.patch,
> MAPREDUCE-2953.patch
>
>
> [~Karams] ran into this multiple times. MR JobClient crashes immediately.
> {code}
> 11/09/08 10:52:35 INFO mapreduce.JobSubmitter: number of splits:2094
> 11/09/08 10:52:36 INFO mapred.YARNRunner: AppMaster capability = memory: 2048,
> 11/09/08 10:52:36 INFO mapred.YARNRunner: Command to launch container for
> ApplicationMaster is : $JAVA_HOME/bin/java -Dhadoop.root.logger=INFO,console
> -Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1315478927026 1
> 1>/stdout 2>/stderr
> 11/09/08 10:52:36 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1315478927026_1 to ResourceManager
> 11/09/08 10:52:36 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user/gridperf/.staging/job_1315478927026_0001
> RemoteTrace:
> at Local Trace:
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
> failed to run job
> at
> org.apache.hadoop.yarn.factories.impl.pb.YarnRemoteExceptionFactoryPBImpl.createYarnRemoteException(YarnRemoteExceptionFactoryPBImpl.java:39)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:47)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:250)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:377)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.examples.RandomWriter.run(RandomWriter.java:283)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.examples.RandomWriter.main(RandomWriter.java:294)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
> }
> {code}
> The client crashes due to a race in RM.
> Because the client fails, it immediately removes the staged files which in
> turn makes the MR AM itself to crash due to failed localization on the NM.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2690) Construct the web page for default scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100873#comment-13100873 ] Hudson commented on MAPREDUCE-2690: --- Integrated in Hadoop-Mapreduce-trunk-Commit #866 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/866/]) MAPREDUCE-2690. Web-page for FifoScheduler. Contributed by Eric Payne. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166958 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java > Construct the web page for default scheduler > > > Key: MAPREDUCE-2690 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2690 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0 > > Attachments: DefSchedulerScreenshot.2011-09-08.png.jpg, > DefaultSchedulerPage.jpg, MAPREDUCE-2690.2011-09-02.patch, > MAPREDUCE-2690.2011-09-05.patch, MAPREDUCE-2690.2011-09-08.patch > > > Currently, the web page for default scheduler reads as "Under construction". > This is a long known issue, but could not find a tracking ticket. Hence > opening one. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2937) Errors in Application failures are not shown in the client trace.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2937: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Mahadev! > Errors in Application failures are not shown in the client trace. > - > > Key: MAPREDUCE-2937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2937 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2937.patch, MAPREDUCE-2937.patch, > MAPREDUCE-2937.patch > > > The client side does not show enough information on why the job failed. Here > is step to reproduce it: > 1) set the scheduler to be capacity scheduler with queues a, b > 2) submit a job to a queue that is not a,b > The job just fails without saying why it failed. We should have enough trace > log at the client side to let the user know why it failed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2690) Construct the web page for default scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100867#comment-13100867 ] Hudson commented on MAPREDUCE-2690: --- Integrated in Hadoop-Common-trunk-Commit #855 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/855/]) MAPREDUCE-2690. Web-page for FifoScheduler. Contributed by Eric Payne. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166958 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java > Construct the web page for default scheduler > > > Key: MAPREDUCE-2690 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2690 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0 > > Attachments: DefSchedulerScreenshot.2011-09-08.png.jpg, > DefaultSchedulerPage.jpg, MAPREDUCE-2690.2011-09-02.patch, > MAPREDUCE-2690.2011-09-05.patch, MAPREDUCE-2690.2011-09-08.patch > > > Currently, the web page for default scheduler reads as "Under construction". > This is a long known issue, but could not find a tracking ticket. Hence > opening one. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2690) Construct the web page for default scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100869#comment-13100869 ] Hudson commented on MAPREDUCE-2690: --- Integrated in Hadoop-Hdfs-trunk-Commit #932 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/932/]) MAPREDUCE-2690. Web-page for FifoScheduler. Contributed by Eric Payne. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166958 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java > Construct the web page for default scheduler > > > Key: MAPREDUCE-2690 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2690 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0 > > Attachments: DefSchedulerScreenshot.2011-09-08.png.jpg, > DefaultSchedulerPage.jpg, MAPREDUCE-2690.2011-09-02.patch, > MAPREDUCE-2690.2011-09-05.patch, MAPREDUCE-2690.2011-09-08.patch > > > Currently, the web page for default scheduler reads as "Under construction". > This is a long known issue, but could not find a tracking ticket. Hence > opening one. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2690) Construct the web page for default scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2690: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Eric! > Construct the web page for default scheduler > > > Key: MAPREDUCE-2690 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2690 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0 > > Attachments: DefSchedulerScreenshot.2011-09-08.png.jpg, > DefaultSchedulerPage.jpg, MAPREDUCE-2690.2011-09-02.patch, > MAPREDUCE-2690.2011-09-05.patch, MAPREDUCE-2690.2011-09-08.patch > > > Currently, the web page for default scheduler reads as "Under construction". > This is a long known issue, but could not find a tracking ticket. Hence > opening one. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2958) wordcount succeeds but doesn't output file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100861#comment-13100861 ] Thomas Graves commented on MAPREDUCE-2958: -- It looks like the mapred-defaults.xml isn't included in any of the mapreduce-client jar files. It used to be in the client-core jar. That is what was setting the default number of reduces to 1. The code current defaults to 0 if not specified. job.numReduceTasks = job.conf.getInt(MRJobConfig.NUM_REDUCES, 0); > wordcount succeeds but doesn't output file > -- > > Key: MAPREDUCE-2958 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2958 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.0 > > > I have been running wordcount out of the 23 examples jar. It says it > succeeds but doesn't actually output a file. > hadoop jar > examples/hadoop-mapreduce-0.23.0-SNAPSHOT/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar > wordcount input output2 > input file is really basic: > fdksajl > dlkfsajlfljda;j > kldfsjallj > test > one > two > test -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2864) Renaming of configuration property names in yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2864: - Resolution: Fixed Status: Resolved (was: Patch Available) This was a biggie, thanks for being patient Robert - I just committed this. > Renaming of configuration property names in yarn > > > Key: MAPREDUCE-2864 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2864 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver, mrv2, nodemanager, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0, 0.24.0 > > Attachments: MR-2864-0.23-v10.patch, MR-2864-trunk-v10.patch, > MR-2864-v1.patch, MR-2864-v2.patch, MR-2864-v3.patch, MR-2864-v4.patch, > MR-2864-v5.patch, MR-2864-v6.patch, MR-2864-v7.patch, MR-2864-v8.patch, > MR-2864-v9.patch, update.pl, yarnConfigs.csv > > > Now that YARN has been put in to trunk we should do something similar to > MAPREDUCE-849. We should go back and look at all of the configurations that > have been added in and rename them as needed to be consistent and subdivided > by component. > # We should use all lowercase in the config names. e.g., we should use > appsmanager instead of appsManager etc. > # history server config names should be prefixed with mapreduce instead of > yarn. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2864) Renaming of configuration property names in yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100858#comment-13100858 ] Hudson commented on MAPREDUCE-2864: --- Integrated in Hadoop-Mapreduce-trunk-Commit #865 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/865/]) MAPREDUCE-2864. Normalize configuration variable names for YARN. Contributed by Robert Evans. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166955 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedProgressSplitsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AMConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/ExponentiallySmoothedTaskRuntimeEstimator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/client/ClientHSSecurityInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main
[jira] [Commented] (MAPREDUCE-2864) Renaming of configuration property names in yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100854#comment-13100854 ] Hudson commented on MAPREDUCE-2864: --- Integrated in Hadoop-Common-trunk-Commit #854 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/854/]) MAPREDUCE-2864. Normalize configuration variable names for YARN. Contributed by Robert Evans. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166955 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedProgressSplitsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AMConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/ExponentiallySmoothedTaskRuntimeEstimator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/client/ClientHSSecurityInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/
[jira] [Commented] (MAPREDUCE-2864) Renaming of configuration property names in yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100855#comment-13100855 ] Hudson commented on MAPREDUCE-2864: --- Integrated in Hadoop-Hdfs-trunk-Commit #931 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/931/]) MAPREDUCE-2864. Normalize configuration variable names for YARN. Contributed by Robert Evans. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166955 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/INSTALL * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/WrappedProgressSplitsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AMConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/ExponentiallySmoothedTaskRuntimeEstimator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/MRConstants.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/client/ClientHSSecurityInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/
[jira] [Commented] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100845#comment-13100845
]
Hadoop QA commented on MAPREDUCE-2954:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493709/MR2954_1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/669//console
This message is automatically generated.
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(
[jira] [Updated] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2843: - Status: Open (was: Patch Available) @Abhijit, Can you please look at the test failure in the patch? > [MR-279] Node entries on the RM UI are not sortable > --- > > Key: MAPREDUCE-2843 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Ramya Sunil >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2843- scr shot.jpg, MAPREDUCE-2843.patch, > MAPREDUCE-2843.patch > > > The nodemanager entries on the RM UI is not sortable unlike the other web > pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-2954:
--
Attachment: MR2954_1.patch
compareTo and equals should not be synchronized in ApplicationAttemptId
The patch fixes this +
- ads synchronization for other methods in ApplicationId and ContainerId.
- hashCode, equals, toString, compareTo implemented for ApplicationId,
ApplicationAttemptId and ContainerId - so that the backing ProtoBuf object is
not serialized for each of these calls.
- Moved these methods up one level (AppId, AppAttemptId, CId changed to
abstract classes)
The tests are kindof lame. Can be dropped if they're not required.
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated MAPREDUCE-2954:
--
Fix Version/s: 0.24.0
Affects Version/s: 0.24.0
Status: Patch Available (was: Open)
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR2954_1.patch
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2413) TaskTracker should handle disk failures at both startup and runtime
[ https://issues.apache.org/jira/browse/MAPREDUCE-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100835#comment-13100835 ] Eli Collins commented on MAPREDUCE-2413: Another testing question - was this tested in conjunction with a mapred health checker script? > TaskTracker should handle disk failures at both startup and runtime > --- > > Key: MAPREDUCE-2413 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2413 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: task-controller, tasktracker >Affects Versions: 0.20.204.0 >Reporter: Bharath Mundlapudi >Assignee: Ravi Gummadi > Fix For: 0.20.204.0 > > Attachments: MR-2413.v0.1.patch, MR-2413.v0.2.patch, > MR-2413.v0.3.patch, MR-2413.v0.patch > > > At present, TaskTracker doesn't handle disk failures properly both at startup > and runtime. > (1) Currently TaskTracker doesn't come up if any of the mapred-local-dirs is > on a bad disk. TaskTracker should ignore that particular mapred-local-dir and > start up and use only the remaining good mapred-local-dirs. > (2) If a disk goes bad while TaskTracker is running, currently TaskTracker > doesn't do anything special. This results in either >(a) TaskTracker continues to "try to use that bad disk" and this results > in lots of task failures and possibly job failures(because of multiple TTs > having bad disks) and eventually these TTs getting graylisted for all jobs. > And this needs manual restart of TT with modified configuration of > mapred-local-dirs avoiding the bad disk. OR >(b) Health check script identifying the disk as bad and the TT gets > blacklisted. And this also needs manual restart of TT with modified > configuration of mapred-local-dirs avoiding the bad disk. > This JIRA is to make TaskTracker more fault-tolerant to disk failures solving > (1) and (2). i.e. TT should start even if at least one of the > mapred-local-dirs is on a good disk and TT should adjust its in-memory list > of mapred-local-dirs and avoid using bad mapred-local-dirs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-434) local map-reduce job limited to single reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100834#comment-13100834 ] Forest Tan commented on MAPREDUCE-434: -- Can't wait for this patch to be submitted... > local map-reduce job limited to single reducer > -- > > Key: MAPREDUCE-434 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-434 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: local job tracker >Reporter: Yoram Arnon >Assignee: Aaron Kimball >Priority: Minor > Attachments: MAPREDUCE-434.2.patch, MAPREDUCE-434.3.patch, > MAPREDUCE-434.4.patch, MAPREDUCE-434.5.patch, MAPREDUCE-434.6.patch, > MAPREDUCE-434.patch > > > when mapred.job.tracker is set to 'local', my setNumReduceTasks call is > ignored, and the number of reduce tasks is set at 1. > This prevents me from locally debugging my partition function, which tries to > partition based on the number of reduce tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2749) [MR-279] NM registers with RM even before it starts various servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100832#comment-13100832 ] Hadoop QA commented on MAPREDUCE-2749: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493701/MAPREDUCE-2749-v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/668//console This message is automatically generated. > [MR-279] NM registers with RM even before it starts various servers > --- > > Key: MAPREDUCE-2749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2749 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Thomas Graves > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2749-v2.patch, MAPREDUCE-2749-v3.patch, > MAPREDUCE-2749-v4.patch, MAPREDUCE-2749.patch > > > In case NM eventually fails to start the ContainerManager server because of > say a port clash, RM will have to wait for expiry to detect the NM crash. > It is desirable to make NM register with RM only after it can start all of > its components successfully. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100828#comment-13100828 ] Hadoop QA commented on MAPREDUCE-2843: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493702/MAPREDUCE-2843.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/667//console This message is automatically generated. > [MR-279] Node entries on the RM UI are not sortable > --- > > Key: MAPREDUCE-2843 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Ramya Sunil >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2843- scr shot.jpg, MAPREDUCE-2843.patch, > MAPREDUCE-2843.patch > > > The nodemanager entries on the RM UI is not sortable unlike the other web > pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2676) MR-279: JobHistory Job page needs reformatted
[ https://issues.apache.org/jira/browse/MAPREDUCE-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100818#comment-13100818 ] Hadoop QA commented on MAPREDUCE-2676: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493619/MR-2676-trunk-V5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/666//console This message is automatically generated. > MR-279: JobHistory Job page needs reformatted > - > > Key: MAPREDUCE-2676 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2676 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.0 > > Attachments: MR-2676-trunk-V2.patch, MR-2676-trunk-V3.patch, > MR-2676-trunk-V4.patch, MR-2676-trunk-V5.patch, MR-2676-v1.patch > > > The Job page, The Maps page and the Reduces page for the job history server > needs to be reformatted. > The Job Overview needs to add in the User, a link to the Job Conf, and the > Job ACLs > It also needs Submitted at, launched at, and finished at, depending on how > they relates to Started and Elapsed. > In the attempts table we need to remove the new and the running columns > In the tasks table we need to remove progress, pending, and running columns > and add in a failed count column > We also need to investigate what it would take to add in setup and cleanup > statistics. Perhaps these should be more generally Application Master > statistics and links. > The Maps page and Reduces page should have the progress column removed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2843: - Attachment: MAPREDUCE-2843.patch Reuploading Abhijit's patch to kick of hudson CI on the right patch. > [MR-279] Node entries on the RM UI are not sortable > --- > > Key: MAPREDUCE-2843 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Ramya Sunil >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2843- scr shot.jpg, MAPREDUCE-2843.patch, > MAPREDUCE-2843.patch > > > The nodemanager entries on the RM UI is not sortable unlike the other web > pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2843: - Status: Open (was: Patch Available) > [MR-279] Node entries on the RM UI are not sortable > --- > > Key: MAPREDUCE-2843 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Ramya Sunil >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2843- scr shot.jpg, MAPREDUCE-2843.patch, > MAPREDUCE-2843.patch > > > The nodemanager entries on the RM UI is not sortable unlike the other web > pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2843) [MR-279] Node entries on the RM UI are not sortable
[ https://issues.apache.org/jira/browse/MAPREDUCE-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2843: - Status: Patch Available (was: Open) > [MR-279] Node entries on the RM UI are not sortable > --- > > Key: MAPREDUCE-2843 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2843 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Ramya Sunil >Assignee: Abhijit Suresh Shingate > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2843- scr shot.jpg, MAPREDUCE-2843.patch, > MAPREDUCE-2843.patch > > > The nodemanager entries on the RM UI is not sortable unlike the other web > pages. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2749) [MR-279] NM registers with RM even before it starts various servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2749: - Status: Patch Available (was: Open) > [MR-279] NM registers with RM even before it starts various servers > --- > > Key: MAPREDUCE-2749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2749 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Thomas Graves > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2749-v2.patch, MAPREDUCE-2749-v3.patch, > MAPREDUCE-2749-v4.patch, MAPREDUCE-2749.patch > > > In case NM eventually fails to start the ContainerManager server because of > say a port clash, RM will have to wait for expiry to detect the NM crash. > It is desirable to make NM register with RM only after it can start all of > its components successfully. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2749) [MR-279] NM registers with RM even before it starts various servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-2749: - Attachment: MAPREDUCE-2749-v4.patch > [MR-279] NM registers with RM even before it starts various servers > --- > > Key: MAPREDUCE-2749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2749 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Thomas Graves > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2749-v2.patch, MAPREDUCE-2749-v3.patch, > MAPREDUCE-2749-v4.patch, MAPREDUCE-2749.patch > > > In case NM eventually fails to start the ContainerManager server because of > say a port clash, RM will have to wait for expiry to detect the NM crash. > It is desirable to make NM register with RM only after it can start all of > its components successfully. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2691) Finish up the cleanup of distributed cache file resources and related tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2691: - Status: Open (was: Patch Available) Cancelling patch so that the issues can be addressed. > Finish up the cleanup of distributed cache file resources and related tests. > > > Key: MAPREDUCE-2691 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2691 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Amol Kekre >Assignee: Siddharth Seth > Fix For: 0.23.0, 0.24.0 > > Attachments: MR2691_1.patch, MR2691_2.patch, MR2691_3.patch > > > Implement cleanup of distributed cache file resources -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
TestMRJobs hangs waiting to connect to history server. -- Key: MAPREDUCE-2963 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Mahadev konar Priority: Critical Fix For: 0.23.0 TestMRJobs is hanging waiting to connect to history server. I will post the logs next. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2963) TestMRJobs hangs waiting to connect to history server.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100807#comment-13100807
]
Mahadev konar commented on MAPREDUCE-2963:
--
{code}
Call:
protocol=org.apache.hadoop.yarn.proto.MRClientProtocol.MRClientProtocolService.BlockingInterface,
method=getTaskReports
2011-09-08 17:10:50,808 INFO mapred.ClientServiceDelegate
(ClientServiceDelegate.java:invoke(242)) - Failed to contact AM for job
job_1315526960327_0001 Will retry..
Call:
protocol=org.apache.hadoop.yarn.proto.ClientRMProtocol.ClientRMProtocolService.BlockingInterface,
method=getApplicationReport
2011-09-08 17:10:50,808 INFO mapred.ClientServiceDelegate
(ClientServiceDelegate.java:getProxy(199)) - Application state is completed.
Redirecting to job history server null
Call:
protocol=org.apache.hadoop.yarn.proto.MRClientProtocol.MRClientProtocolService.BlockingInterface,
method=getTaskReports
2011-09-08 17:10:50,809 INFO mapred.ClientServiceDelegate
(ClientServiceDelegate.java:invoke(242)) - Failed to contact AM for job
job_1315526960327_0001 Will retry..
Call:
protocol=org.apache.hadoop.yarn.proto.ClientRMProtocol.ClientRMProtocolService.BlockingInterface,
method=getApplicationReport
2011-09-08 17:10:50,809 INFO mapred.ClientServiceDelegate
(ClientServiceDelegate.java:getProxy(199)) - Application state is completed.
Redirecting to job history server null
Call:
protocol=org.apache.hadoop.yarn.proto.MRClientProtocol.MRClientProtocolService.BlockingInterface,
method=getTaskReports
2011-09-08 17:10:50,809 INFO mapred.ClientServiceDelegate
(ClientServiceDelegate.java:invoke(242)) - Failed to contact AM for job
job_1315526960
{code}
This is trace I see when the first job completes in the TestMRJobs.
> TestMRJobs hangs waiting to connect to history server.
> --
>
> Key: MAPREDUCE-2963
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2963
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Mahadev konar
>Priority: Critical
> Fix For: 0.23.0
>
>
> TestMRJobs is hanging waiting to connect to history server. I will post the
> logs next.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool size for container launching in the application master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100803#comment-13100803 ] Hadoop QA commented on MAPREDUCE-2961: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493691/MAPREDUCE-2961.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//console This message is automatically generated. > Increase the default threadpool size for container launching in the > application master. > --- > > Key: MAPREDUCE-2961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2961.patch > > > Currently the default threadpool size is 10 for launching containers in > ContainerLauncherImpl. We should increase that to 100 for a reasonable > default, so that container launching is not backed up by a small thread pool > size. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2864) Renaming of configuration property names in yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100802#comment-13100802
]
Mahadev konar commented on MAPREDUCE-2864:
--
Minor nit, just noticed this in the patch in yarn-default.xml:
{code}
- nodemanager.aux.service.mapreduce.shuffle.class
- org.apache.hadoop.mapred.ShuffleHandler
+mapreduce.job.jar
+
+
--->
{code}
I think we should remove this property name (mapreduce.job.jar) in
yarn-default.xml. This can be done when committing, no need to upload a new
patch.
> Renaming of configuration property names in yarn
>
>
> Key: MAPREDUCE-2864
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2864
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver, mrv2, nodemanager, resourcemanager
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Fix For: 0.23.0, 0.24.0
>
> Attachments: MR-2864-0.23-v10.patch, MR-2864-trunk-v10.patch,
> MR-2864-v1.patch, MR-2864-v2.patch, MR-2864-v3.patch, MR-2864-v4.patch,
> MR-2864-v5.patch, MR-2864-v6.patch, MR-2864-v7.patch, MR-2864-v8.patch,
> MR-2864-v9.patch, update.pl, yarnConfigs.csv
>
>
> Now that YARN has been put in to trunk we should do something similar to
> MAPREDUCE-849. We should go back and look at all of the configurations that
> have been added in and rename them as needed to be consistent and subdivided
> by component.
> # We should use all lowercase in the config names. e.g., we should use
> appsmanager instead of appsManager etc.
> # history server config names should be prefixed with mapreduce instead of
> yarn.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2954) Deadlock in NM with threads racing for ApplicationAttemptId
[
https://issues.apache.org/jira/browse/MAPREDUCE-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth reassigned MAPREDUCE-2954:
-
Assignee: Siddharth Seth
> Deadlock in NM with threads racing for ApplicationAttemptId
> ---
>
> Key: MAPREDUCE-2954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2954
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Siddharth Seth
>Priority: Critical
> Fix For: 0.23.0
>
>
> Found this:
> {code}
> Java stack information for the threads listed above:
> ===
> "Thread-45":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:797)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:360)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:355)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:619)
> "Thread-30":
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.getApplicationId(ApplicationAttemptIdPBImpl.java:101)
> - waiting to lock <0xb6a443a0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:144)
> - locked <0xb6a43ba0> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationAttemptIdPBImpl.compareTo(ApplicationAttemptIdPBImpl.java:31)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:215)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ContainerIdPBImpl.compareTo(ContainerIdPBImpl.java:34)
> at
> java.util.concurrent.ConcurrentSkipListMap.doRemove(ConcurrentSkipListMap.java:1078)
> at
> java.util.concurrent.ConcurrentSkipListMap.remove(ConcurrentSkipListMap.java:1673)
> at
> java.util.concurrent.ConcurrentSkipListMap$Iter.remove(ConcurrentSkipListMap.java:2256)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.getNodeStatus(NodeStatusUpdaterImpl.java:223)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.access$300(NodeStatusUpdaterImpl.java:62)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:262)
> Found 1 deadlock.
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool size for container launching in the application master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated MAPREDUCE-2961: -- Summary: Increase the default threadpool size for container launching in the application master. (was: Increase the default threadpool fize for container launching in the application master.) > Increase the default threadpool size for container launching in the > application master. > --- > > Key: MAPREDUCE-2961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2961.patch > > > Currently the default threadpool size is 10 for launching containers in > ContainerLauncherImpl. We should increase that to 100 for a reasonable > default, so that container launching is not backed up by a small thread pool > size. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2940) Build fails with ant 1.7.0 but works with 1.8.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100791#comment-13100791 ] Hadoop QA commented on MAPREDUCE-2940: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493690/MAPREDUCE-2940.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/664//console This message is automatically generated. > Build fails with ant 1.7.0 but works with 1.8.0 > --- > > Key: MAPREDUCE-2940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2940 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 > Environment: Ubuntu 11.04; Sun JDK 1.6.0_26; Ant 1.8.2; Ant 1.7.0 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2940.patch > > > contrib builds fail when using Ant 1.7. > build.xml calls build.xml in contrib, which calls block-forensics build, > which in turn uses build-contrib. > The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool fize for container launching in the application master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2961: - Status: Patch Available (was: Open) > Increase the default threadpool fize for container launching in the > application master. > --- > > Key: MAPREDUCE-2961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2961.patch > > > Currently the default threadpool size is 10 for launching containers in > ContainerLauncherImpl. We should increase that to 100 for a reasonable > default, so that container launching is not backed up by a small thread pool > size. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool fize for container launching in the application master.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-2961: - Attachment: MAPREDUCE-2961.patch Trivial patch to increase the number of threads in the launcher so that we do not bottleneck on container launching. > Increase the default threadpool fize for container launching in the > application master. > --- > > Key: MAPREDUCE-2961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2961.patch > > > Currently the default threadpool size is 10 for launching containers in > ContainerLauncherImpl. We should increase that to 100 for a reasonable > default, so that container launching is not backed up by a small thread pool > size. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2962) Map percentage complete calculation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100786#comment-13100786 ] Todd Lipcon commented on MAPREDUCE-2962: There were a couple of "lost map output" errors here, but it's strange that the web UI doesn't show the tasks moving back from Complete to Pending/Running > Map percentage complete calculation incorrect > - > > Key: MAPREDUCE-2962 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2962 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.203.0 >Reporter: Todd Lipcon >Priority: Minor > > For some reason I have a job with 5000 map tasks, all of which are complete, > and it's reporting 99.98% complete. The client logs show "map 100%" and then > going back down to "map 99%". -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2940) Build fails with ant 1.7.0 but works with 1.8.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated MAPREDUCE-2940: Status: Patch Available (was: Open) > Build fails with ant 1.7.0 but works with 1.8.0 > --- > > Key: MAPREDUCE-2940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2940 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 > Environment: Ubuntu 11.04; Sun JDK 1.6.0_26; Ant 1.8.2; Ant 1.7.0 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2940.patch > > > contrib builds fail when using Ant 1.7. > build.xml calls build.xml in contrib, which calls block-forensics build, > which in turn uses build-contrib. > The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2962) Map percentage complete calculation incorrect
Map percentage complete calculation incorrect - Key: MAPREDUCE-2962 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2962 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.203.0 Reporter: Todd Lipcon Priority: Minor For some reason I have a job with 5000 map tasks, all of which are complete, and it's reporting 99.98% complete. The client logs show "map 100%" and then going back down to "map 99%". -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2940) Build fails with ant 1.7.0 but works with 1.8.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated MAPREDUCE-2940: Attachment: MAPREDUCE-2940.patch > Build fails with ant 1.7.0 but works with 1.8.0 > --- > > Key: MAPREDUCE-2940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2940 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0 > Environment: Ubuntu 11.04; Sun JDK 1.6.0_26; Ant 1.8.2; Ant 1.7.0 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2940.patch > > > contrib builds fail when using Ant 1.7. > build.xml calls build.xml in contrib, which calls block-forensics build, > which in turn uses build-contrib. > The inheritAll=true overrides the basedir in ant 1.7.0 but not in 1.8.0. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
