[jira] [Commented] (YARN-1200) Provide a central view for rack topologies
[ https://issues.apache.org/jira/browse/YARN-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768157#comment-13768157 ] Steve Loughran commented on YARN-1200: -- I don't think we should rush to add a broadcast mechanism, esp. as multicasts and the like don't work on public cloud infrastructures # The heartbeats can include a field declaring when the topology was last updated; AMs that care about it could then ask for the information # The HA RM depends on ZK, so we could just add the ability of the RM to publish topo information to ZK; AMs and anything else that cares could just listen to that bit of the ZK tree > Provide a central view for rack topologies > -- > > Key: YARN-1200 > URL: https://issues.apache.org/jira/browse/YARN-1200 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Harsh J > > It appears that with YARN, any AM (such as the MRv2 AM) that tries to do > rack-info-based work, will need to resolve racks locally rather than get rack > info from YARN directly: > https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#L1054 > and its use of a simple implementation of > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java > This is a regression, as we've traditionally only had users maintain rack > mappings and its associated script on a single master role node (JobTracker), > not at every compute node. Task spawning hosts have never done/needed rack > resolution of their own. > It is silly to have to maintain rack configs and their changes on all nodes. > We should have the RM host a stable interface service so that there's only a > single view of the topology across the cluster, and document for AMs to use > that instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1197) Support increasing resources of an allocated container
[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768180#comment-13768180 ] Wangda Tan commented on YARN-1197: -- Here is the tentative plan: To make it simple, we can start with only support increase container resource when container state is "ACQUIRED", because when container is running, we need find a way to inform NM to update the quato of a running process, which will affect lots of parts. # Add a field in ResourceRequest, like set/getExistingContainerId() to indicate the resource will be added to existing container # When the ask-list submitted to YarnScheduler, it will be normalized: ** When existingContainerId set *** Check if container with "existingContainerId" in state ACQUIRED, abondon this request if container not exist OR state not ACQUIRED. *** Check if capability of this container + asked capability > scheduler.maximum-allocation-mb, if yes, normalize ask capability *** Other fields will be ignored except "capability", and the "priority/host" will use the existing container's priority/host *** Update resource requests in AppSchedulingInfo # Add "increase existing container resource" support in LeafQueue of capa/fifo scheduler, and FSLeafQueue in fair scheduler. We can treat "increase existing container resource request" as same as other resource request except, ** Only "node-local" available ** Don't create new container # After we successfully allocated resource to an existing container, we need refresh its container token, send back to AM. AM can use the new container to launch processes via NM. Any thoughts? I wish I could help to move this ahead if we can reach an agreement on a rough idea. :) > Support increasing resources of an allocated container > -- > > Key: YARN-1197 > URL: https://issues.apache.org/jira/browse/YARN-1197 > Project: Hadoop YARN > Issue Type: Task > Components: api, nodemanager, resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Wangda Tan > > Currently, YARN cannot support merge several containers in one node to a big > container, which can make us incrementally ask resources, merge them to a > bigger one, and launch our processes. The user scenario is described in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1200) Provide a central view for rack topologies
[ https://issues.apache.org/jira/browse/YARN-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768195#comment-13768195 ] Harsh J commented on YARN-1200: --- A reasonable regression-fixing first step is to match that of the HDFS functionality: Each NameNode (and NameNode alone) needs the rack resolution script, not all the DNs. > Provide a central view for rack topologies > -- > > Key: YARN-1200 > URL: https://issues.apache.org/jira/browse/YARN-1200 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Harsh J > > It appears that with YARN, any AM (such as the MRv2 AM) that tries to do > rack-info-based work, will need to resolve racks locally rather than get rack > info from YARN directly: > https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#L1054 > and its use of a simple implementation of > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java > This is a regression, as we've traditionally only had users maintain rack > mappings and its associated script on a single master role node (JobTracker), > not at every compute node. Task spawning hosts have never done/needed rack > resolution of their own. > It is silly to have to maintain rack configs and their changes on all nodes. > We should have the RM host a stable interface service so that there's only a > single view of the topology across the cluster, and document for AMs to use > that instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1200) Provide a central view for rack topologies
[ https://issues.apache.org/jira/browse/YARN-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768228#comment-13768228 ] Junping Du commented on YARN-1200: -- bq. A reasonable regression-fixing first step is to match that of the HDFS functionality: Each NameNode (and NameNode alone) needs the rack resolution script, not all the DNs. Right. I also agree this is the first step we should address. However, I think this is not a simple fix that we should quickly make a decision. The reason is as below: Yarn, unlike MRV1, actually separate two layers of scheduling (resource scheduling and task scheduling) out for simplicity: For resource scheduling, AM translate tasks (TaskAttempt) from Job into ContainerRequest and later encode to ResourceRequests and send to RM scheduler for container allocation. For task scheduling, AM schedule map tasks based on locality on allocated containers. For the first step, the AM resolve topology to create several (typically 3: node, rack and ANY/*) ResourceRequest from one ContainerRequest asked for specific nodes. The second step is decided by AM only which means AM has to understand the topology at least for some specific nodes. Although we may get rid of resolving topology in the first step by simplifying ResourceRequest between AM and RM, i.e. sending only 1 (node) ResourceRequest instead of 3 RRs, we still need to resolve it in second step. Given we cannot get rid of resolve topology in AM, we may prefer cache instead of running the script on each node. I have a draft proposal below: - Setup a cache in AM - In response come back from AMRM heartbeat, AM can get topology info on related nodes (nodes in request and assignedContainer) and add into cache - Remove all RackResolver call in AM side with accessing cache - If cache missing in sending node's ResourceRequest for rack, resolve its rack location to something unusual (like:UNKNOWN), then RM will replace with correct rack info. Heartbeat back will fill the cache later. - The cache is not only be filled but also be refreshed. So if topology changes and aware in RM side, the changes may not updated immediately to all AMs, but gradually be updated to related AMs (request node's resource or get assigned containers). Thoughts? > Provide a central view for rack topologies > -- > > Key: YARN-1200 > URL: https://issues.apache.org/jira/browse/YARN-1200 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Harsh J > > It appears that with YARN, any AM (such as the MRv2 AM) that tries to do > rack-info-based work, will need to resolve racks locally rather than get rack > info from YARN directly: > https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#L1054 > and its use of a simple implementation of > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java > This is a regression, as we've traditionally only had users maintain rack > mappings and its associated script on a single master role node (JobTracker), > not at every compute node. Task spawning hosts have never done/needed rack > resolution of their own. > It is silly to have to maintain rack configs and their changes on all nodes. > We should have the RM host a stable interface service so that there's only a > single view of the topology across the cluster, and document for AMs to use > that instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1027) Implement RMHAProtocolService
[ https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1027: --- Attachment: yarn-1027-9.patch Including YARN-1192 (RM should enter HAServiceState#STOPPING on stop()). > Implement RMHAProtocolService > - > > Key: YARN-1027 > URL: https://issues.apache.org/jira/browse/YARN-1027 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Karthik Kambatla > Attachments: test-yarn-1027.patch, yarn-1027-1.patch, > yarn-1027-2.patch, yarn-1027-3.patch, yarn-1027-4.patch, yarn-1027-5.patch, > yarn-1027-6.patch, yarn-1027-7.patch, yarn-1027-7.patch, yarn-1027-8.patch, > yarn-1027-9.patch, yarn-1027-including-yarn-1098-3.patch, > yarn-1027-in-rm-poc.patch > > > Implement existing HAServiceProtocol from Hadoop common. This protocol is the > single point of interaction between the RM and HA clients/services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1202) Verify RM HA works in secure clusters
Karthik Kambatla created YARN-1202: -- Summary: Verify RM HA works in secure clusters Key: YARN-1202 URL: https://issues.apache.org/jira/browse/YARN-1202 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Karthik Kambatla Assignee: Karthik Kambatla We should verify RM HA works in secure clusters, and fix any issues that we come across. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1192) Update HAServiceState to STOPPING on RM#stop()
[ https://issues.apache.org/jira/browse/YARN-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved YARN-1192. Resolution: Duplicate yarn-1027-9.patch includes a fix for this. > Update HAServiceState to STOPPING on RM#stop() > -- > > Key: YARN-1192 > URL: https://issues.apache.org/jira/browse/YARN-1192 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > Post HADOOP-9945, we should update HAServiceState in RMHAProtocolService to > STOPPING on stop(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1027) Implement RMHAProtocolService
[ https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1027: --- Attachment: yarn-1027-9.patch > Implement RMHAProtocolService > - > > Key: YARN-1027 > URL: https://issues.apache.org/jira/browse/YARN-1027 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Karthik Kambatla > Attachments: test-yarn-1027.patch, yarn-1027-1.patch, > yarn-1027-2.patch, yarn-1027-3.patch, yarn-1027-4.patch, yarn-1027-5.patch, > yarn-1027-6.patch, yarn-1027-7.patch, yarn-1027-7.patch, yarn-1027-8.patch, > yarn-1027-9.patch, yarn-1027-including-yarn-1098-3.patch, > yarn-1027-in-rm-poc.patch > > > Implement existing HAServiceProtocol from Hadoop common. This protocol is the > single point of interaction between the RM and HA clients/services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1027) Implement RMHAProtocolService
[ https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-1027: --- Attachment: (was: yarn-1027-9.patch) > Implement RMHAProtocolService > - > > Key: YARN-1027 > URL: https://issues.apache.org/jira/browse/YARN-1027 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Karthik Kambatla > Attachments: test-yarn-1027.patch, yarn-1027-1.patch, > yarn-1027-2.patch, yarn-1027-3.patch, yarn-1027-4.patch, yarn-1027-5.patch, > yarn-1027-6.patch, yarn-1027-7.patch, yarn-1027-7.patch, yarn-1027-8.patch, > yarn-1027-9.patch, yarn-1027-including-yarn-1098-3.patch, > yarn-1027-in-rm-poc.patch > > > Implement existing HAServiceProtocol from Hadoop common. This protocol is the > single point of interaction between the RM and HA clients/services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1170) yarn proto definitions should specify package as 'hadoop.yarn'
[ https://issues.apache.org/jira/browse/YARN-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768338#comment-13768338 ] Hudson commented on YARN-1170: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4421 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4421/]) YARN-1170. YARN & MapReduce proto definitions fixed to specify protobuf package as hadoop.yarn and hadoop.mapreduce respectively. Contibuted by Binglin Chang. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523652) * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/MRClientProtocol.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/proto/mr_service_protos.proto * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/proto/HSAdminRefreshProtocol.proto * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationmaster_protocol.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/containermanagement_protocol.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/resourcemanager_administration_protocol.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/yarn_server_resourcemanager_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/ResourceTracker.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/proto/LocalizationProtocol.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/proto/yarn_server_nodemanager_service_protos.proto > yarn proto definitions should specify package as 'hadoop.yarn' > -- > > Key: YARN-1170 > URL: https://issues.apache.org/jira/browse/YARN-1170 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun C Murthy >Assignee: Binglin Chang >Priority: Blocker > Fix For: 2.1.1-beta > > Attachments: YARN-1170.v1.patch, YARN-1170.v1.patch > > > yarn proto definitions should specify package as 'hadoop.yarn' similar to > protos with 'hadoop.common' & 'hadoop.hdfs' in Common & HDFS respectively. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1027) Implement RMHAProtocolService
[
https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768365#comment-13768365
]
Hadoop QA commented on YARN-1027:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603331/yarn-1027-9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1937//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1937//console
This message is automatically generated.
> Implement RMHAProtocolService
> -
>
> Key: YARN-1027
> URL: https://issues.apache.org/jira/browse/YARN-1027
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Karthik Kambatla
> Attachments: test-yarn-1027.patch, yarn-1027-1.patch,
> yarn-1027-2.patch, yarn-1027-3.patch, yarn-1027-4.patch, yarn-1027-5.patch,
> yarn-1027-6.patch, yarn-1027-7.patch, yarn-1027-7.patch, yarn-1027-8.patch,
> yarn-1027-9.patch, yarn-1027-including-yarn-1098-3.patch,
> yarn-1027-in-rm-poc.patch
>
>
> Implement existing HAServiceProtocol from Hadoop common. This protocol is the
> single point of interaction between the RM and HA clients/services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1199) Make NM/RM Versions Available
[ https://issues.apache.org/jira/browse/YARN-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-1199: -- Attachment: YARN-1199.patch > Make NM/RM Versions Available > - > > Key: YARN-1199 > URL: https://issues.apache.org/jira/browse/YARN-1199 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: YARN-1199.patch > > > Now as we have the NM and RM Versions available, we can display the YARN > version of nodes running in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1027) Implement RMHAProtocolService
[
https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768492#comment-13768492
]
Karthik Kambatla commented on YARN-1027:
{quote}
// TODO (YARN-1192): Update this post addition of STOPPING state to
Is this still needed since I see that HADOOP-9945 is already committed?
{quote}
Included in the latest patch.
bq. Looks like not all the memory was reclaimed upon Active->Standby. One thing
to check would be if the memory keeps increasing after every transition for
multiple transitions.
The previous set of numbers were taken immediately after the transitions. I
suspect it takes a little bit for GC to reclaim. I have done a new set of
experiments:
# Started the RM, transitioned to Active (active-1) and ran 10 pi jobs.
# Transitioned to standby (Standby-1) -> Active (active-2) -> Standby
(standby-2) -> Active (active-3) -> Standby (standby-3), waiting for some time
after transitioning to standby to let the heap reach steady state. The memory
consumption is as below. The first number is the number of objects, and the
second number is bytes.
{noformat}
active-1 417253 50697304
standby-1 387763 46063712
active-2 404628 48989688
standby-2 391583 46265168
active-3 381511 47552168
standby-3 379597 45919224
{noformat}
Interestingly, standby-3 bytes is the smallest. I think the objects are being
reclaimed; however, the actual heap size depends on if/when/what GC kicks in. A
more thorough evaluation can be done as part of YARN-1125 and YARN-1139.
bq. Do you plan to test this patch under secure cluster?
Filed YARN-1202 to verify RM HA in secure clusters. Given we haven't made any
security-related changes, I don't think we are causing any regressions.
> Implement RMHAProtocolService
> -
>
> Key: YARN-1027
> URL: https://issues.apache.org/jira/browse/YARN-1027
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bikas Saha
>Assignee: Karthik Kambatla
> Attachments: test-yarn-1027.patch, yarn-1027-1.patch,
> yarn-1027-2.patch, yarn-1027-3.patch, yarn-1027-4.patch, yarn-1027-5.patch,
> yarn-1027-6.patch, yarn-1027-7.patch, yarn-1027-7.patch, yarn-1027-8.patch,
> yarn-1027-9.patch, yarn-1027-including-yarn-1098-3.patch,
> yarn-1027-in-rm-poc.patch
>
>
> Implement existing HAServiceProtocol from Hadoop common. This protocol is the
> single point of interaction between the RM and HA clients/services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1027) Implement RMHAProtocolService
[ https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768502#comment-13768502 ] Karthik Kambatla commented on YARN-1027: Update: Leaving the RM in standby mode led to further reduce in heap size - standby-3 376600 45734624 > Implement RMHAProtocolService > - > > Key: YARN-1027 > URL: https://issues.apache.org/jira/browse/YARN-1027 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Karthik Kambatla > Attachments: test-yarn-1027.patch, yarn-1027-1.patch, > yarn-1027-2.patch, yarn-1027-3.patch, yarn-1027-4.patch, yarn-1027-5.patch, > yarn-1027-6.patch, yarn-1027-7.patch, yarn-1027-7.patch, yarn-1027-8.patch, > yarn-1027-9.patch, yarn-1027-including-yarn-1098-3.patch, > yarn-1027-in-rm-poc.patch > > > Implement existing HAServiceProtocol from Hadoop common. This protocol is the > single point of interaction between the RM and HA clients/services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1028) Add FailoverProxyProvider like capability to RMProxy
[ https://issues.apache.org/jira/browse/YARN-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768509#comment-13768509 ] Karthik Kambatla commented on YARN-1028: Hi [~devaraj.k], did you get a chance to work on this? If you are otherwise busy, I ll be happy to take a stab. > Add FailoverProxyProvider like capability to RMProxy > > > Key: YARN-1028 > URL: https://issues.apache.org/jira/browse/YARN-1028 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Devaraj K > > RMProxy layer currently abstracts RM discovery and implements it by looking > up service information from configuration. Motivated by HDFS and using > existing classes from Common, we can add failover proxy providers that may > provide RM discovery in extensible ways. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1203) Need to be able to disable 'hadoop.ssl.enabled' explicitly
Yesha Vora created YARN-1203: Summary: Need to be able to disable 'hadoop.ssl.enabled' explicitly Key: YARN-1203 URL: https://issues.apache.org/jira/browse/YARN-1203 Project: Hadoop YARN Issue Type: Bug Reporter: Yesha Vora Need to add support to disable 'hadoop.ssl.enabled' for MR jobs. A job should be able to run on http protocol by setting 'hadoop.ssl.enabled' property at job level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-1203) Need to be able to disable 'hadoop.ssl.enabled' explicitly
[ https://issues.apache.org/jira/browse/YARN-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi reassigned YARN-1203: --- Assignee: Omkar Vinit Joshi > Need to be able to disable 'hadoop.ssl.enabled' explicitly > -- > > Key: YARN-1203 > URL: https://issues.apache.org/jira/browse/YARN-1203 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi > > Need to add support to disable 'hadoop.ssl.enabled' for MR jobs. > A job should be able to run on http protocol by setting 'hadoop.ssl.enabled' > property at job level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768549#comment-13768549
]
Hadoop QA commented on YARN-1001:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603372/YARN-1001.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1938//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1938//console
This message is automatically generated.
> YARN should provide per application-type and state statistics
> -
>
> Key: YARN-1001
> URL: https://issues.apache.org/jira/browse/YARN-1001
> Project: Hadoop YARN
> Issue Type: Task
> Components: api
>Affects Versions: 2.1.0-beta
>Reporter: Srimanth Gunturi
>Assignee: Zhijie Shen
>Priority: Blocker
> Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch,
> YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch
>
>
> In Ambari we plan to show for MR2 the number of applications finished,
> running, waiting, etc. It would be efficient if YARN could provide per
> application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768464#comment-13768464 ] Xuan Gong commented on YARN-1001: - May need to update the patch with the latest trunk. RMServerUtils. createApplicationState() has been changed > YARN should provide per application-type and state statistics > - > > Key: YARN-1001 > URL: https://issues.apache.org/jira/browse/YARN-1001 > Project: Hadoop YARN > Issue Type: Task > Components: api >Affects Versions: 2.1.0-beta >Reporter: Srimanth Gunturi >Assignee: Zhijie Shen >Priority: Blocker > Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, > YARN-1001.4.patch, YARN-1001.5.patch > > > In Ambari we plan to show for MR2 the number of applications finished, > running, waiting, etc. It would be efficient if YARN could provide per > application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1200) Provide a central view for rack topologies
[ https://issues.apache.org/jira/browse/YARN-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved YARN-1200. --- Resolution: Duplicate Duplicate of YARN-435. > Provide a central view for rack topologies > -- > > Key: YARN-1200 > URL: https://issues.apache.org/jira/browse/YARN-1200 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Harsh J > > It appears that with YARN, any AM (such as the MRv2 AM) that tries to do > rack-info-based work, will need to resolve racks locally rather than get rack > info from YARN directly: > https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#L1054 > and its use of a simple implementation of > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java > This is a regression, as we've traditionally only had users maintain rack > mappings and its associated script on a single master role node (JobTracker), > not at every compute node. Task spawning hosts have never done/needed rack > resolution of their own. > It is silly to have to maintain rack configs and their changes on all nodes. > We should have the RM host a stable interface service so that there's only a > single view of the topology across the cluster, and document for AMs to use > that instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1001: -- Attachment: YARN-1001.6.patch Rebase with the latest trunk > YARN should provide per application-type and state statistics > - > > Key: YARN-1001 > URL: https://issues.apache.org/jira/browse/YARN-1001 > Project: Hadoop YARN > Issue Type: Task > Components: api >Affects Versions: 2.1.0-beta >Reporter: Srimanth Gunturi >Assignee: Zhijie Shen >Priority: Blocker > Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, > YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch > > > In Ambari we plan to show for MR2 the number of applications finished, > running, waiting, etc. It would be efficient if YARN could provide per > application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1203) Need to be able to disable 'hadoop.ssl.enabled' explicitly
[ https://issues.apache.org/jira/browse/YARN-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768620#comment-13768620 ] Omkar Vinit Joshi commented on YARN-1203: - Today if SSL is enabled for RM/NM/NN/DN then it gets enabled even for Map reduce application master. Potentially there is security problem here * We can't allow MR users to gain access to keystore file * Also Certificates issued by MR-AM can not be trusted by end user just like the one issued by RM/NM. Also to keystore and truststore only yarn and hdfs users should have access to (RW-RW yarn superuser). Explicitly disabling SSL for MR AM (hadoop.ssl.enabled=false) > Need to be able to disable 'hadoop.ssl.enabled' explicitly > -- > > Key: YARN-1203 > URL: https://issues.apache.org/jira/browse/YARN-1203 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi > > Need to add support to disable 'hadoop.ssl.enabled' for MR jobs. > A job should be able to run on http protocol by setting 'hadoop.ssl.enabled' > property at job level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1027) Implement RMHAProtocolService
[ https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768614#comment-13768614 ] Hudson commented on YARN-1027: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4423 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4423/]) YARN-1027. Implement RMHAProtocolService (Karthik Kambatla via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523750) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMHAProtocolService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMContainerTokenSecretManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationmasterservice/TestApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java > Implement RMHAProtocolService > - > > Key: YARN-1027 > URL: https://issues.apache.org/jira/browse/YARN-1027 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Karthik Kambatla > Fix For: 2.3.0 > > Attachments: test-yarn-1027.patch, yarn-1027-1.patch, > yarn-1027-2.patch, yarn-1027-3.patch, yarn-1027-4.patch, yarn-1027-5.patch, > yarn-1027-6.patch, yarn-1027-7.patch, yarn-1027-7.patch, yarn-1027-8.patch, > yarn-1027-9.patch, yarn-1027-including-yarn-1098-3.patch, > yarn-1027-in-rm-poc.patch > > > Implement existing HAServiceProtocol from Hadoop common. This protocol is the > single point of interaction between the RM and HA clients/services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1125) Add shutdown support to non-service RM components
[ https://issues.apache.org/jira/browse/YARN-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768578#comment-13768578 ] Bikas Saha commented on YARN-1125: -- We are currently doing this by "deleting" these components by dereferencing them. It might happen these do not get GC'd and we are leaking memory. In that case we this jira will become important because we will need to implement a close/shutdown functionality to all of them. > Add shutdown support to non-service RM components > - > > Key: YARN-1125 > URL: https://issues.apache.org/jira/browse/YARN-1125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Karthik Kambatla > > The ResourceManager has certain non-service components like the Scheduler. > While transitioning to standby, these components should be completely turned > off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1027) Implement RMHAProtocolService
[ https://issues.apache.org/jira/browse/YARN-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768586#comment-13768586 ] Bikas Saha commented on YARN-1027: -- +1. I am committing this patch since there is no memory leak for existing RM and this patch sets up the framework to perform failover. We need to continue to watch the memory leak situation and if we find that its a real issue then YARN-1125 will become a blocker for YARN-149. > Implement RMHAProtocolService > - > > Key: YARN-1027 > URL: https://issues.apache.org/jira/browse/YARN-1027 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Karthik Kambatla > Attachments: test-yarn-1027.patch, yarn-1027-1.patch, > yarn-1027-2.patch, yarn-1027-3.patch, yarn-1027-4.patch, yarn-1027-5.patch, > yarn-1027-6.patch, yarn-1027-7.patch, yarn-1027-7.patch, yarn-1027-8.patch, > yarn-1027-9.patch, yarn-1027-including-yarn-1098-3.patch, > yarn-1027-in-rm-poc.patch > > > Implement existing HAServiceProtocol from Hadoop common. This protocol is the > single point of interaction between the RM and HA clients/services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1205) Not able to run sleep job
Omkar Vinit Joshi created YARN-1205: --- Summary: Not able to run sleep job Key: YARN-1205 URL: https://issues.apache.org/jira/browse/YARN-1205 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Omkar Vinit Joshi Priority: Blocker Not able to run sleep job on latest trunk bin/yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-353) Add Zookeeper-based store implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-353: -- Attachment: YARN-353.16.patch > Add Zookeeper-based store implementation for RMStateStore > - > > Key: YARN-353 > URL: https://issues.apache.org/jira/browse/YARN-353 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Hitesh Shah >Assignee: Karthik Kambatla > Attachments: YARN-353.10.patch, YARN-353.11.patch, YARN-353.12.patch, > yarn-353-12-wip.patch, YARN-353.13.patch, YARN-353.14.patch, > YARN-353.15.patch, YARN-353.16.patch, YARN-353.1.patch, YARN-353.2.patch, > YARN-353.3.patch, YARN-353.4.patch, YARN-353.5.patch, YARN-353.6.patch, > YARN-353.7.patch, YARN-353.8.patch, YARN-353.9.patch > > > Add store that write RM state data to ZK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1068) Add admin support for HA operations
[
https://issues.apache.org/jira/browse/YARN-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-1068:
---
Description: Support HA admin operations to facilitate transitioning the RM
to Active and Standby states. (was: To transitionTo{Active,Standby} etc. we
should support admin operations the same way DFS does.)
> Add admin support for HA operations
> ---
>
> Key: YARN-1068
> URL: https://issues.apache.org/jira/browse/YARN-1068
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Labels: ha
> Attachments: yarn-1068-1.patch, yarn-1068-2.patch,
> yarn-1068-prelim.patch
>
>
> Support HA admin operations to facilitate transitioning the RM to Active and
> Standby states.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1068) Add admin support for HA operations
[
https://issues.apache.org/jira/browse/YARN-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-1068:
---
Attachment: yarn-1068-2.patch
Patch rebased on trunk and documentation of configs.
> Add admin support for HA operations
> ---
>
> Key: YARN-1068
> URL: https://issues.apache.org/jira/browse/YARN-1068
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-1068-1.patch, yarn-1068-2.patch,
> yarn-1068-prelim.patch
>
>
> To transitionTo{Active,Standby} etc. we should support admin operations the
> same way DFS does.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-353) Add Zookeeper-based store implementation for RMStateStore
[ https://issues.apache.org/jira/browse/YARN-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768725#comment-13768725 ] Karthik Kambatla commented on YARN-353: --- The latest patch addresses Hitesh's latest set of review comments. Some of the spurious looking changes in TestRMRestart or to fix lines longer than 80 chars. I tested this on a pseudo-dist cluster: # Started the RM using ZKRMStateStore # Ran jobs - all ZKnodes are created as expected # Restarted the RM several times while a job is running - the job succeeds. > Add Zookeeper-based store implementation for RMStateStore > - > > Key: YARN-353 > URL: https://issues.apache.org/jira/browse/YARN-353 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Hitesh Shah >Assignee: Karthik Kambatla > Attachments: YARN-353.10.patch, YARN-353.11.patch, YARN-353.12.patch, > yarn-353-12-wip.patch, YARN-353.13.patch, YARN-353.14.patch, > YARN-353.15.patch, YARN-353.16.patch, YARN-353.1.patch, YARN-353.2.patch, > YARN-353.3.patch, YARN-353.4.patch, YARN-353.5.patch, YARN-353.6.patch, > YARN-353.7.patch, YARN-353.8.patch, YARN-353.9.patch > > > Add store that write RM state data to ZK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-353) Add Zookeeper-based store implementation for RMStateStore
[
https://issues.apache.org/jira/browse/YARN-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768773#comment-13768773
]
Hadoop QA commented on YARN-353:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603419/YARN-353.16.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1939//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1939//console
This message is automatically generated.
> Add Zookeeper-based store implementation for RMStateStore
> -
>
> Key: YARN-353
> URL: https://issues.apache.org/jira/browse/YARN-353
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Hitesh Shah
>Assignee: Karthik Kambatla
> Attachments: YARN-353.10.patch, YARN-353.11.patch, YARN-353.12.patch,
> yarn-353-12-wip.patch, YARN-353.13.patch, YARN-353.14.patch,
> YARN-353.15.patch, YARN-353.16.patch, YARN-353.1.patch, YARN-353.2.patch,
> YARN-353.3.patch, YARN-353.4.patch, YARN-353.5.patch, YARN-353.6.patch,
> YARN-353.7.patch, YARN-353.8.patch, YARN-353.9.patch
>
>
> Add store that write RM state data to ZK
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1205) Not able to run sleep job
[ https://issues.apache.org/jira/browse/YARN-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-1205: Description: Not able to run sleep job on latest trunk bin/yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar output :- Picked up _JAVA_OPTIONS: -Djava.awt.headless=true java.lang.NoClassDefFoundError: junit/framework/TestCase at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) at java.lang.ClassLoader.defineClass(ClassLoader.java:615) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:60) at org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:54) at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) ... 20 more An example program must be given as the first argument. Valid program names are: was: Not able to run sleep job on latest trunk bin/yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar > Not able to run sleep job > - > > Key: YARN-1205 > URL: https://issues.apache.org/jira/browse/YARN-1205 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Omkar Vinit Joshi >Priority: Blocker > > Not able to run sleep job on latest trunk > bin/yarn jar > ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar > output :- > Picked up _JAVA_OPTIONS: -Djava.awt.headless=true > java.lang.NoClassDefFoundError: junit/framework/TestCase > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:60) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:54) > at > org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(U
[jira] [Reopened] (YARN-1190) enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb
[ https://issues.apache.org/jira/browse/YARN-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li reopened YARN-1190: --- > enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to > be less than yarn.app.mapreduce.am.resource.mb > > > Key: YARN-1190 > URL: https://issues.apache.org/jira/browse/YARN-1190 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Siqi Li >Priority: Minor > > Since there is no reducer, the memory allocated to reducer is irrelevant to > enable uber mode of a job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1123) [YARN-321] Adding ContainerReport and Protobuf implementation
[ https://issues.apache.org/jira/browse/YARN-1123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768841#comment-13768841 ] Mayank Bansal commented on YARN-1123: - We have COntainer class under org.apache.hadoop.yarn.api.records which we can use for the report. Can you please update your concerns, why we can't do that? Thanks, Mayank > [YARN-321] Adding ContainerReport and Protobuf implementation > - > > Key: YARN-1123 > URL: https://issues.apache.org/jira/browse/YARN-1123 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Like YARN-978, we need some client-oriented class to expose the container > history info. Neither Container nor RMContainer is the right one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1190) enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb
[ https://issues.apache.org/jira/browse/YARN-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li resolved YARN-1190. --- Resolution: Fixed > enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to > be less than yarn.app.mapreduce.am.resource.mb > > > Key: YARN-1190 > URL: https://issues.apache.org/jira/browse/YARN-1190 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Siqi Li >Priority: Minor > > Since there is no reducer, the memory allocated to reducer is irrelevant to > enable uber mode of a job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1068) Add admin support for HA operations
[
https://issues.apache.org/jira/browse/YARN-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768799#comment-13768799
]
Hadoop QA commented on YARN-1068:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603434/yarn-1068-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.client.api.impl.TestYarnClient
org.apache.hadoop.yarn.client.api.impl.TestNMClient
org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.TestResourceManager
org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
org.apache.hadoop.yarn.server.resourcemanager.applicationmasterservice.TestApplicationMasterService
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
org.apache.hadoop.yarn.server.resourcemanager.TestRM
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
org.apache.hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1940//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1940//console
This message is automatically generated.
> Add admin support for HA operations
> ---
>
> Key: YARN-1068
> URL: https://issues.apache.org/jira/browse/YARN-1068
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Labels: ha
> Attachments: yarn-1068-1.patch, yarn-1068-2.patch,
> yarn-1068-prelim.patch
>
>
> Support HA admin operations to facilitate transitioning the RM to Active and
> Standby states.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-987) Adding History Service to use Store and converting Historydata to Report
[ https://issues.apache.org/jira/browse/YARN-987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-987: --- Attachment: YARN-987-2.patch Thanks [~zjshen] for the review. I have updated with your comments. And we are using containerhistorydata in the patch. Thanks, Mayank > Adding History Service to use Store and converting Historydata to Report > > > Key: YARN-987 > URL: https://issues.apache.org/jira/browse/YARN-987 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: YARN-987-1.patch, YARN-987-2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1203) Application Manager UI does not appear with Https enabled
[ https://issues.apache.org/jira/browse/YARN-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated YARN-1203: - Summary: Application Manager UI does not appear with Https enabled (was: Need to be able to disable 'hadoop.ssl.enabled' explicitly) > Application Manager UI does not appear with Https enabled > - > > Key: YARN-1203 > URL: https://issues.apache.org/jira/browse/YARN-1203 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi > > Need to add support to disable 'hadoop.ssl.enabled' for MR jobs. > A job should be able to run on http protocol by setting 'hadoop.ssl.enabled' > property at job level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1190) enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb
[ https://issues.apache.org/jira/browse/YARN-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated YARN-1190: -- Attachment: YARN-1190_v1.patch.txt > enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to > be less than yarn.app.mapreduce.am.resource.mb > > > Key: YARN-1190 > URL: https://issues.apache.org/jira/browse/YARN-1190 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.0.6-alpha >Reporter: Siqi Li >Priority: Minor > Attachments: YARN-1190_v1.patch.txt > > > Since there is no reducer, the memory allocated to reducer is irrelevant to > enable uber mode of a job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1190) enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb
[ https://issues.apache.org/jira/browse/YARN-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li resolved YARN-1190. --- Resolution: Fixed > enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to > be less than yarn.app.mapreduce.am.resource.mb > > > Key: YARN-1190 > URL: https://issues.apache.org/jira/browse/YARN-1190 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.0.6-alpha >Reporter: Siqi Li >Priority: Minor > Attachments: YARN-1190_v1.patch.txt > > > Since there is no reducer, the memory allocated to reducer is irrelevant to > enable uber mode of a job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1200) Provide a central view for rack topologies
[ https://issues.apache.org/jira/browse/YARN-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768826#comment-13768826 ] Junping Du commented on YARN-1200: -- Hi Vinod, thanks for pointing to YARN-435. I like the idea to use ClientRMProtocol (now should be ApplicationClientProtocol) in AM which make things much simpler. However, does that JIRA also cover replacing RackResolver.resolve() and maintain a topology cache in AM? I don't see any discussions on this but just want to understand the scope of that JIRA. Thanks! > Provide a central view for rack topologies > -- > > Key: YARN-1200 > URL: https://issues.apache.org/jira/browse/YARN-1200 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Harsh J > > It appears that with YARN, any AM (such as the MRv2 AM) that tries to do > rack-info-based work, will need to resolve racks locally rather than get rack > info from YARN directly: > https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java#L1054 > and its use of a simple implementation of > https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java > This is a regression, as we've traditionally only had users maintain rack > mappings and its associated script on a single master role node (JobTracker), > not at every compute node. Task spawning hosts have never done/needed rack > resolution of their own. > It is silly to have to maintain rack configs and their changes on all nodes. > We should have the RM host a stable interface service so that there's only a > single view of the topology across the cluster, and document for AMs to use > that instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-353) Add Zookeeper-based store implementation for RMStateStore
[
https://issues.apache.org/jira/browse/YARN-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768721#comment-13768721
]
Karthik Kambatla commented on YARN-353:
---
Thanks again for the detailed review, [~hitesh]. Sorry for the slow turnaround
on this.
{quote}
{code}
} catch (Exception e) {
e.printStackTrace();
Assert.fail("ZKRMStateStore Session restore failed");
}
{code}
{quote}
bq. Don't think there is any need to catch the exception. The unit test will
fail if the exception is not caught.
Agree. The difference is in what the test reports shows this as - Failure or
Error. The general rule I try to follow is - errors encountered in the specific
code we are testing should be marked failures. I am not particular about this
though, let me know if you would like me to make that change as well. The
updated patch addresses some of these inconsistencies as well.
> Add Zookeeper-based store implementation for RMStateStore
> -
>
> Key: YARN-353
> URL: https://issues.apache.org/jira/browse/YARN-353
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Hitesh Shah
>Assignee: Karthik Kambatla
> Attachments: YARN-353.10.patch, YARN-353.11.patch, YARN-353.12.patch,
> yarn-353-12-wip.patch, YARN-353.13.patch, YARN-353.14.patch,
> YARN-353.15.patch, YARN-353.16.patch, YARN-353.1.patch, YARN-353.2.patch,
> YARN-353.3.patch, YARN-353.4.patch, YARN-353.5.patch, YARN-353.6.patch,
> YARN-353.7.patch, YARN-353.8.patch, YARN-353.9.patch
>
>
> Add store that write RM state data to ZK
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-1001: -- Attachment: YARN-1001.7.patch Addressed Vinod's comments. Now if applicationTypes is not provided, we count all applications of any application type. > YARN should provide per application-type and state statistics > - > > Key: YARN-1001 > URL: https://issues.apache.org/jira/browse/YARN-1001 > Project: Hadoop YARN > Issue Type: Task > Components: api >Affects Versions: 2.1.0-beta >Reporter: Srimanth Gunturi >Assignee: Zhijie Shen >Priority: Blocker > Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, > YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch > > > In Ambari we plan to show for MR2 the number of applications finished, > running, waiting, etc. It would be efficient if YARN could provide per > application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-979) [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol
[ https://issues.apache.org/jira/browse/YARN-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768831#comment-13768831 ] Mayank Bansal commented on YARN-979: Thanks [~zjshen] for updating the patch. Let me reassign this to me and take it forward. Looks like we don't need ApplicationAttemptReport and ContainerReport and those are already covered in seprate jiras. Working on this now. Thanks, Mayank > [YARN-321] Add more APIs related to ApplicationAttempt and Container in > ApplicationHistoryProtocol > -- > > Key: YARN-979 > URL: https://issues.apache.org/jira/browse/YARN-979 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Zhijie Shen > Attachments: YARN-979-1.patch, YARN-979.2.patch > > > ApplicationHistoryProtocol should have the following APIs as well: > * getApplicationAttemptReport > * getApplicationAttempts > * getContainerReport > * getContainers > The corresponding request and response classes need to be added as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-979) [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol
[ https://issues.apache.org/jira/browse/YARN-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal reassigned YARN-979: -- Assignee: Mayank Bansal (was: Zhijie Shen) > [YARN-321] Add more APIs related to ApplicationAttempt and Container in > ApplicationHistoryProtocol > -- > > Key: YARN-979 > URL: https://issues.apache.org/jira/browse/YARN-979 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: YARN-979-1.patch, YARN-979.2.patch > > > ApplicationHistoryProtocol should have the following APIs as well: > * getApplicationAttemptReport > * getApplicationAttempts > * getContainerReport > * getContainers > The corresponding request and response classes need to be added as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-987) Adding History Service to use Store and converting Historydata to Report
[
https://issues.apache.org/jira/browse/YARN-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768832#comment-13768832
]
Hadoop QA commented on YARN-987:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603448/YARN-987-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1941//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1941//console
This message is automatically generated.
> Adding History Service to use Store and converting Historydata to Report
>
>
> Key: YARN-987
> URL: https://issues.apache.org/jira/browse/YARN-987
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Mayank Bansal
>Assignee: Mayank Bansal
> Attachments: YARN-987-1.patch, YARN-987-2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1205) Not able to run sleep job
[ https://issues.apache.org/jira/browse/YARN-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768830#comment-13768830 ] Jason Lowe commented on YARN-1205: -- I suspect this is caused by HADOOP-9935. > Not able to run sleep job > - > > Key: YARN-1205 > URL: https://issues.apache.org/jira/browse/YARN-1205 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Omkar Vinit Joshi >Priority: Blocker > > Not able to run sleep job on latest trunk > bin/yarn jar > ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar > output :- > Picked up _JAVA_OPTIONS: -Djava.awt.headless=true > java.lang.NoClassDefFoundError: junit/framework/TestCase > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:60) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:54) > at > org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 20 more > An example program must be given as the first argument. > Valid program names are: -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1205) Not able to run sleep job
[ https://issues.apache.org/jira/browse/YARN-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768885#comment-13768885 ] André Kelpe commented on YARN-1205: --- Shouldn't junit be included in the tests jar then, instead of having it on the cluster itself? Every other program brings the libs it needs with it. Why wouldn't that be the case here? > Not able to run sleep job > - > > Key: YARN-1205 > URL: https://issues.apache.org/jira/browse/YARN-1205 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Omkar Vinit Joshi >Priority: Blocker > > Not able to run sleep job on latest trunk > bin/yarn jar > ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar > output :- > Picked up _JAVA_OPTIONS: -Djava.awt.headless=true > java.lang.NoClassDefFoundError: junit/framework/TestCase > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:60) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:54) > at > org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 20 more > An example program must be given as the first argument. > Valid program names are: -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1123) [YARN-321] Adding ContainerReport and Protobuf implementation
[ https://issues.apache.org/jira/browse/YARN-1123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768925#comment-13768925 ] Zhijie Shen commented on YARN-1123: --- Currently, there're two existing choices: records.Container and RMContainer. 1. RMContainer is obviously not suitable as it is the internal stuff of RM 2. Record is the proto that is used during the communication between AM and RM. It contains the info for RM to allocate container. The info is also not user oriented, such as ContainerToken. Therefore, like ApplicationReport and ApplicationAttemptReport, we need a proto class to wrap the user oriented info and expose it to the users. > [YARN-321] Adding ContainerReport and Protobuf implementation > - > > Key: YARN-1123 > URL: https://issues.apache.org/jira/browse/YARN-1123 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Like YARN-978, we need some client-oriented class to expose the container > history info. Neither Container nor RMContainer is the right one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768948#comment-13768948
]
Hadoop QA commented on YARN-1001:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603470/YARN-1001.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1942//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1942//console
This message is automatically generated.
> YARN should provide per application-type and state statistics
> -
>
> Key: YARN-1001
> URL: https://issues.apache.org/jira/browse/YARN-1001
> Project: Hadoop YARN
> Issue Type: Task
> Components: api
>Affects Versions: 2.1.0-beta
>Reporter: Srimanth Gunturi
>Assignee: Zhijie Shen
>Priority: Blocker
> Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch,
> YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch
>
>
> In Ambari we plan to show for MR2 the number of applications finished,
> running, waiting, etc. It would be efficient if YARN could provide per
> application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-1123) [YARN-321] Adding ContainerReport and Protobuf implementation
[ https://issues.apache.org/jira/browse/YARN-1123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal reassigned YARN-1123: --- Assignee: Mayank Bansal (was: Zhijie Shen) > [YARN-321] Adding ContainerReport and Protobuf implementation > - > > Key: YARN-1123 > URL: https://issues.apache.org/jira/browse/YARN-1123 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > > Like YARN-978, we need some client-oriented class to expose the container > history info. Neither Container nor RMContainer is the right one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1123) [YARN-321] Adding ContainerReport and Protobuf implementation
[ https://issues.apache.org/jira/browse/YARN-1123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768956#comment-13768956 ] Mayank Bansal commented on YARN-1123: - ok, let me then do it. Thanks, Mayank > [YARN-321] Adding ContainerReport and Protobuf implementation > - > > Key: YARN-1123 > URL: https://issues.apache.org/jira/browse/YARN-1123 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Like YARN-978, we need some client-oriented class to expose the container > history info. Neither Container nor RMContainer is the right one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1205) Not able to run sleep job
[ https://issues.apache.org/jira/browse/YARN-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1205: -- Target Version/s: 2.1.1-beta > Not able to run sleep job > - > > Key: YARN-1205 > URL: https://issues.apache.org/jira/browse/YARN-1205 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Omkar Vinit Joshi >Priority: Blocker > > Not able to run sleep job on latest trunk > bin/yarn jar > ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar > output :- > Picked up _JAVA_OPTIONS: -Djava.awt.headless=true > java.lang.NoClassDefFoundError: junit/framework/TestCase > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:60) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:54) > at > org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 20 more > An example program must be given as the first argument. > Valid program names are: -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1206) Container logs link is broken on RM web UI after application finished
Jian He created YARN-1206: - Summary: Container logs link is broken on RM web UI after application finished Key: YARN-1206 URL: https://issues.apache.org/jira/browse/YARN-1206 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He When container is running, its logs link works properly, but after the application is finished, the link shows 'Container does not exist.' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-451) Add more metrics to RM page
[ https://issues.apache.org/jira/browse/YARN-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768989#comment-13768989 ] Sangjin Lee commented on YARN-451: -- I am going to upload a single patch to get your feedback. I wanted to get your feedback first to validate the approach before I break out sub-tasks and post separate patches. This single patch is solely for review, and is not to be merged. > Add more metrics to RM page > --- > > Key: YARN-451 > URL: https://issues.apache.org/jira/browse/YARN-451 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.0.3-alpha >Reporter: Lohit Vijayarenu >Priority: Minor > > ResourceManager webUI shows list of RUNNING applications, but it does not > tell which applications are requesting more resource compared to others. With > cluster running hundreds of applications at once it would be useful to have > some kind of metric to show high-resource usage applications vs low-resource > usage ones. At the minimum showing number of containers is good option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-451) Add more metrics to RM page
[ https://issues.apache.org/jira/browse/YARN-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-451: - Attachment: yarn-451-trunk-20130916.1.patch > Add more metrics to RM page > --- > > Key: YARN-451 > URL: https://issues.apache.org/jira/browse/YARN-451 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.0.3-alpha >Reporter: Lohit Vijayarenu >Priority: Minor > Attachments: yarn-451-trunk-20130916.1.patch > > > ResourceManager webUI shows list of RUNNING applications, but it does not > tell which applications are requesting more resource compared to others. With > cluster running hundreds of applications at once it would be useful to have > some kind of metric to show high-resource usage applications vs low-resource > usage ones. At the minimum showing number of containers is good option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page
[
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-1044:
--
Attachment: yarn-451-trunk-20130916.1.patch
> used/min/max resources do not display info in the scheduler page
>
>
> Key: YARN-1044
> URL: https://issues.apache.org/jira/browse/YARN-1044
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Priority: Minor
> Labels: newbie
> Attachments: screenshot.png, yarn-1044-20130815.3.patch,
> yarn-1044.patch, yarn-1044.patch
>
>
> Go to the scheduler page in RM, and click any queue to display the detailed
> info. You'll find that none of the resources entries (used, min, or max)
> would display values.
> It is because the values contain brackets ("<" and ">") and are not properly
> html-escaped.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1044) used/min/max resources do not display info in the scheduler page
[
https://issues.apache.org/jira/browse/YARN-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated YARN-1044:
--
Attachment: (was: yarn-451-trunk-20130916.1.patch)
> used/min/max resources do not display info in the scheduler page
>
>
> Key: YARN-1044
> URL: https://issues.apache.org/jira/browse/YARN-1044
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, scheduler
>Affects Versions: 2.0.5-alpha
>Reporter: Sangjin Lee
>Priority: Minor
> Labels: newbie
> Attachments: screenshot.png, yarn-1044-20130815.3.patch,
> yarn-1044.patch, yarn-1044.patch
>
>
> Go to the scheduler page in RM, and click any queue to display the detailed
> info. You'll find that none of the resources entries (used, min, or max)
> would display values.
> It is because the values contain brackets ("<" and ">") and are not properly
> html-escaped.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-649) Make container logs available over HTTP in plain text
[ https://issues.apache.org/jira/browse/YARN-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768995#comment-13768995 ] Jian He commented on YARN-649: -- The container logs link on web UI is broken after this check in. opened YARN-1206 > Make container logs available over HTTP in plain text > - > > Key: YARN-649 > URL: https://issues.apache.org/jira/browse/YARN-649 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.3.0 > > Attachments: YARN-649-2.patch, YARN-649-3.patch, YARN-649-4.patch, > YARN-649-5.patch, YARN-649-6.patch, YARN-649-7.patch, YARN-649.patch, > YARN-752-1.patch > > > It would be good to make container logs available over the REST API for > MAPREDUCE-4362 and so that they can be accessed programatically in general. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-451) Add more metrics to RM page
[ https://issues.apache.org/jira/browse/YARN-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-451: - Attachment: in_progress_2x.png A screenshot from this patch. "Total memory" and "Total cores" have been added. > Add more metrics to RM page > --- > > Key: YARN-451 > URL: https://issues.apache.org/jira/browse/YARN-451 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.0.3-alpha >Reporter: Lohit Vijayarenu >Priority: Minor > Attachments: in_progress_2x.png, yarn-451-trunk-20130916.1.patch > > > ResourceManager webUI shows list of RUNNING applications, but it does not > tell which applications are requesting more resource compared to others. With > cluster running hundreds of applications at once it would be useful to have > some kind of metric to show high-resource usage applications vs low-resource > usage ones. At the minimum showing number of containers is good option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1207) AM fails to register if RM restarts within 5s of job submission
Arpit Gupta created YARN-1207: - Summary: AM fails to register if RM restarts within 5s of job submission Key: YARN-1207 URL: https://issues.apache.org/jira/browse/YARN-1207 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.1.0-beta Reporter: Arpit Gupta -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1207) AM fails to register if RM restarts within 5s of job submission
[
https://issues.apache.org/jira/browse/YARN-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769000#comment-13769000
]
Arpit Gupta commented on YARN-1207:
---
Here is the stack trace from the log
{code}
2013-09-16 21:40:36,852 INFO [main] org.apache.hadoop.ipc.Client: Retrying
connect to server: hostname/ip:8030. Already tried 9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1 SECONDS)
2013-09-16 21:40:37,853 INFO [main] org.apache.hadoop.ipc.Client: Retrying
connect to server: hostname/ip:8030. Already tried 10 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1 SECONDS)
2013-09-16 21:40:38,854 INFO [main] org.apache.hadoop.ipc.Client: Retrying
connect to server: hostname/ip:8030. Already tried 11 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1 SECONDS)
2013-09-16 21:40:39,855 INFO [main] org.apache.hadoop.ipc.Client: Retrying
connect to server: hostname/ip:8030. Already tried 12 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1 SECONDS)
2013-09-16 21:40:39,895 ERROR [main]
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hrt_qa (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt appattempt_1379367602708_0001_01
2013-09-16 21:40:39,896 WARN [main] org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt appattempt_1379367602708_0001_01
2013-09-16 21:40:39,896 ERROR [main]
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hrt_qa (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
Password not found for ApplicationAttempt appattempt_1379367602708_0001_01
2013-09-16 21:40:39,899 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Exception while
registering
org.apache.hadoop.security.token.SecretManager$InvalidToken: Password not found
for ApplicationAttempt appattempt_1379367602708_0001_01
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at $Proxy30.registerApplicationMaster(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:150)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:110)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:213)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:790)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1025)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1401)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1483)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndS
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769001#comment-13769001 ] Vinod Kumar Vavilapalli commented on YARN-1001: --- +1, looks good. Checking this in. > YARN should provide per application-type and state statistics > - > > Key: YARN-1001 > URL: https://issues.apache.org/jira/browse/YARN-1001 > Project: Hadoop YARN > Issue Type: Task > Components: api >Affects Versions: 2.1.0-beta >Reporter: Srimanth Gunturi >Assignee: Zhijie Shen >Priority: Blocker > Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, > YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch > > > In Ambari we plan to show for MR2 the number of applications finished, > running, waiting, etc. It would be efficient if YARN could provide per > application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1204) Need to add https port related property in Yarn
Yesha Vora created YARN-1204: Summary: Need to add https port related property in Yarn Key: YARN-1204 URL: https://issues.apache.org/jira/browse/YARN-1204 Project: Hadoop YARN Issue Type: Bug Reporter: Yesha Vora There is no yarn property available to configure https port for Resource manager, nodemanager and history server. Currently, Yarn services uses the port defined for http [defined by 'mapreduce.jobhistory.webapp.address','yarn.nodemanager.webapp.address', 'yarn.resourcemanager.webapp.address'] for running services on https protocol. Yarn should have list of property to assign https port for RM, NM and JHS. It can be like below. yarn.nodemanager.webapp.https.address yarn.resourcemanager.webapp.https.address mapreduce.jobhistory.webapp.https.address -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-1204) Need to add https port related property in Yarn
[ https://issues.apache.org/jira/browse/YARN-1204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi reassigned YARN-1204: --- Assignee: Omkar Vinit Joshi > Need to add https port related property in Yarn > --- > > Key: YARN-1204 > URL: https://issues.apache.org/jira/browse/YARN-1204 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Omkar Vinit Joshi > > There is no yarn property available to configure https port for Resource > manager, nodemanager and history server. Currently, Yarn services uses the > port defined for http [defined by > 'mapreduce.jobhistory.webapp.address','yarn.nodemanager.webapp.address', > 'yarn.resourcemanager.webapp.address'] for running services on https protocol. > Yarn should have list of property to assign https port for RM, NM and JHS. > It can be like below. > yarn.nodemanager.webapp.https.address > yarn.resourcemanager.webapp.https.address > mapreduce.jobhistory.webapp.https.address -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769030#comment-13769030 ] Hudson commented on YARN-1001: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4425 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4425/]) YARN-1001. Added a web-service to get statistics about per application-type per state for consumption by downstream projects. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523855) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ApplicationStatisticsInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/StatisticsItemInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm > YARN should provide per application-type and state statistics > - > > Key: YARN-1001 > URL: https://issues.apache.org/jira/browse/YARN-1001 > Project: Hadoop YARN > Issue Type: Task > Components: api >Affects Versions: 2.1.0-beta >Reporter: Srimanth Gunturi >Assignee: Zhijie Shen >Priority: Blocker > Fix For: 2.1.1-beta > > Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, > YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch > > > In Ambari we plan to show for MR2 the number of applications finished, > running, waiting, etc. It would be efficient if YARN could provide per > application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1190) enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb
[ https://issues.apache.org/jira/browse/YARN-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated YARN-1190: --- Affects Version/s: 2.1.1-beta > enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to > be less than yarn.app.mapreduce.am.resource.mb > > > Key: YARN-1190 > URL: https://issues.apache.org/jira/browse/YARN-1190 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta, 2.0.6-alpha >Reporter: Siqi Li >Priority: Minor > Attachments: YARN-1190_v1.patch.txt > > > Since there is no reducer, the memory allocated to reducer is irrelevant to > enable uber mode of a job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-1208) One of the WebUI Links redirected to http instead https protocol with ssl enabled
Yesha Vora created YARN-1208: Summary: One of the WebUI Links redirected to http instead https protocol with ssl enabled Key: YARN-1208 URL: https://issues.apache.org/jira/browse/YARN-1208 Project: Hadoop YARN Issue Type: Bug Reporter: Yesha Vora One of the webUI links is redirecting to the http link when https is enabled. Open Nodemanager UI (https://nodemanager:50060/node/allContainers) and click on RM HOME link. This link redirects to "http://resourcemanager:port"; instead "https://resourcemanager:port"; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (YARN-1190) enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb
[ https://issues.apache.org/jira/browse/YARN-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li reopened YARN-1190: --- > enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to > be less than yarn.app.mapreduce.am.resource.mb > > > Key: YARN-1190 > URL: https://issues.apache.org/jira/browse/YARN-1190 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.0.6-alpha >Reporter: Siqi Li >Priority: Minor > Attachments: YARN-1190_v1.patch.txt > > > Since there is no reducer, the memory allocated to reducer is irrelevant to > enable uber mode of a job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1190) enabling uber mode with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb
[ https://issues.apache.org/jira/browse/YARN-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated YARN-1190: --- Summary: enabling uber mode with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb (was: enabling uber mde with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb) > enabling uber mode with 0 reducer still requires mapreduce.reduce.memory.mb > to be less than yarn.app.mapreduce.am.resource.mb > - > > Key: YARN-1190 > URL: https://issues.apache.org/jira/browse/YARN-1190 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta, 2.0.6-alpha >Reporter: Siqi Li >Priority: Minor > Attachments: YARN-1190_v1.patch.txt > > > Since there is no reducer, the memory allocated to reducer is irrelevant to > enable uber mode of a job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1190) enabling uber mode with 0 reducer still requires mapreduce.reduce.memory.mb to be less than yarn.app.mapreduce.am.resource.mb
[ https://issues.apache.org/jira/browse/YARN-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769046#comment-13769046 ] Joep Rottinghuis commented on YARN-1190: This may have to be moved to the MAPREDUCE jira project. > enabling uber mode with 0 reducer still requires mapreduce.reduce.memory.mb > to be less than yarn.app.mapreduce.am.resource.mb > - > > Key: YARN-1190 > URL: https://issues.apache.org/jira/browse/YARN-1190 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta, 2.0.6-alpha >Reporter: Siqi Li >Priority: Minor > Attachments: YARN-1190_v1.patch.txt > > > Since there is no reducer, the memory allocated to reducer is irrelevant to > enable uber mode of a job -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1206) Container logs link is broken on RM web UI after application finished
[ https://issues.apache.org/jira/browse/YARN-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1206: -- Labels: 2.1.1-beta (was: ) > Container logs link is broken on RM web UI after application finished > - > > Key: YARN-1206 > URL: https://issues.apache.org/jira/browse/YARN-1206 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Priority: Blocker > Labels: 2.1.1-beta > > When container is running, its logs link works properly, but after the > application is finished, the link shows 'Container does not exist.' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1206) Container logs link is broken on RM web UI after application finished
[ https://issues.apache.org/jira/browse/YARN-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1206: -- Priority: Blocker (was: Major) > Container logs link is broken on RM web UI after application finished > - > > Key: YARN-1206 > URL: https://issues.apache.org/jira/browse/YARN-1206 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Priority: Blocker > > When container is running, its logs link works properly, but after the > application is finished, the link shows 'Container does not exist.' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1206) Container logs link is broken on RM web UI after application finished
[ https://issues.apache.org/jira/browse/YARN-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1206: -- Fix Version/s: 2.1.1-beta > Container logs link is broken on RM web UI after application finished > - > > Key: YARN-1206 > URL: https://issues.apache.org/jira/browse/YARN-1206 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Priority: Blocker > Labels: 2.1.1-beta > Fix For: 2.1.1-beta > > > When container is running, its logs link works properly, but after the > application is finished, the link shows 'Container does not exist.' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1184) ClassCastException is thrown during preemption When a huge job is submitted to a queue B whose resources is used by a job in queueA
[
https://issues.apache.org/jira/browse/YARN-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769067#comment-13769067
]
Chris Douglas commented on YARN-1184:
-
I committed this.
Thanks Bikas for the review.
> ClassCastException is thrown during preemption When a huge job is submitted
> to a queue B whose resources is used by a job in queueA
> ---
>
> Key: YARN-1184
> URL: https://issues.apache.org/jira/browse/YARN-1184
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler, resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: J.Andreina
>Assignee: Chris Douglas
> Fix For: 2.1.1-beta
>
> Attachments: Y1184-0.patch, Y1184-1.patch
>
>
> preemption is enabled.
> Queue = a,b
> a capacity = 30%
> b capacity = 70%
> Step 1: Assign a big job to queue a ( so that job_a will utilize some
> resources from queue b)
> Step 2: Assigne a big job to queue b.
> Following exception is thrown at Resource Manager
> {noformat}
> 2013-09-12 10:42:32,535 ERROR [SchedulingMonitor
> (ProportionalCapacityPreemptionPolicy)] yarn.YarnUncaughtExceptionHandler
> (YarnUncaughtExceptionHandler.java:uncaughtException(68)) - Thread
> Thread[SchedulingMonitor (ProportionalCapacityPreemptionPolicy),5,main] threw
> an Exception.
> java.lang.ClassCastException: java.util.Collections$UnmodifiableSet cannot be
> cast to java.util.NavigableSet
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.getContainersToPreempt(ProportionalCapacityPreemptionPolicy.java:403)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.containerBasedPreemptOrKill(ProportionalCapacityPreemptionPolicy.java:202)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.editSchedule(ProportionalCapacityPreemptionPolicy.java:173)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.invokePolicy(SchedulingMonitor.java:72)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor$PreemptionChecker.run(SchedulingMonitor.java:82)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1184) ClassCastException is thrown during preemption When a huge job is submitted to a queue B whose resources is used by a job in queueA
[
https://issues.apache.org/jira/browse/YARN-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated YARN-1184:
Attachment: Y1184-1.patch
> ClassCastException is thrown during preemption When a huge job is submitted
> to a queue B whose resources is used by a job in queueA
> ---
>
> Key: YARN-1184
> URL: https://issues.apache.org/jira/browse/YARN-1184
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler, resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: J.Andreina
>Assignee: Chris Douglas
> Fix For: 2.1.1-beta
>
> Attachments: Y1184-0.patch, Y1184-1.patch
>
>
> preemption is enabled.
> Queue = a,b
> a capacity = 30%
> b capacity = 70%
> Step 1: Assign a big job to queue a ( so that job_a will utilize some
> resources from queue b)
> Step 2: Assigne a big job to queue b.
> Following exception is thrown at Resource Manager
> {noformat}
> 2013-09-12 10:42:32,535 ERROR [SchedulingMonitor
> (ProportionalCapacityPreemptionPolicy)] yarn.YarnUncaughtExceptionHandler
> (YarnUncaughtExceptionHandler.java:uncaughtException(68)) - Thread
> Thread[SchedulingMonitor (ProportionalCapacityPreemptionPolicy),5,main] threw
> an Exception.
> java.lang.ClassCastException: java.util.Collections$UnmodifiableSet cannot be
> cast to java.util.NavigableSet
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.getContainersToPreempt(ProportionalCapacityPreemptionPolicy.java:403)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.containerBasedPreemptOrKill(ProportionalCapacityPreemptionPolicy.java:202)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.editSchedule(ProportionalCapacityPreemptionPolicy.java:173)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.invokePolicy(SchedulingMonitor.java:72)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor$PreemptionChecker.run(SchedulingMonitor.java:82)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1207) AM fails to register if RM restarts within 5s of job submission
[ https://issues.apache.org/jira/browse/YARN-1207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1207: -- Issue Type: Sub-task (was: Bug) Parent: YARN-128 > AM fails to register if RM restarts within 5s of job submission > --- > > Key: YARN-1207 > URL: https://issues.apache.org/jira/browse/YARN-1207 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Arpit Gupta > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1184) ClassCastException is thrown during preemption When a huge job is submitted to a queue B whose resources is used by a job in queueA
[
https://issues.apache.org/jira/browse/YARN-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769071#comment-13769071
]
Hudson commented on YARN-1184:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4426 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4426/])
YARN-1184. ClassCastException during preemption enforcement. (cdouglas:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1523863)
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
> ClassCastException is thrown during preemption When a huge job is submitted
> to a queue B whose resources is used by a job in queueA
> ---
>
> Key: YARN-1184
> URL: https://issues.apache.org/jira/browse/YARN-1184
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler, resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: J.Andreina
>Assignee: Chris Douglas
> Fix For: 2.1.1-beta
>
> Attachments: Y1184-0.patch, Y1184-1.patch
>
>
> preemption is enabled.
> Queue = a,b
> a capacity = 30%
> b capacity = 70%
> Step 1: Assign a big job to queue a ( so that job_a will utilize some
> resources from queue b)
> Step 2: Assigne a big job to queue b.
> Following exception is thrown at Resource Manager
> {noformat}
> 2013-09-12 10:42:32,535 ERROR [SchedulingMonitor
> (ProportionalCapacityPreemptionPolicy)] yarn.YarnUncaughtExceptionHandler
> (YarnUncaughtExceptionHandler.java:uncaughtException(68)) - Thread
> Thread[SchedulingMonitor (ProportionalCapacityPreemptionPolicy),5,main] threw
> an Exception.
> java.lang.ClassCastException: java.util.Collections$UnmodifiableSet cannot be
> cast to java.util.NavigableSet
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.getContainersToPreempt(ProportionalCapacityPreemptionPolicy.java:403)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.containerBasedPreemptOrKill(ProportionalCapacityPreemptionPolicy.java:202)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.editSchedule(ProportionalCapacityPreemptionPolicy.java:173)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor.invokePolicy(SchedulingMonitor.java:72)
> at
> org.apache.hadoop.yarn.server.resourcemanager.monitor.SchedulingMonitor$PreemptionChecker.run(SchedulingMonitor.java:82)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-1205) Not able to run sleep job
[ https://issues.apache.org/jira/browse/YARN-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved YARN-1205. - Resolution: Not A Problem Resolving since HADOOP-9935 has been reverted. > Not able to run sleep job > - > > Key: YARN-1205 > URL: https://issues.apache.org/jira/browse/YARN-1205 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.1.1-beta >Reporter: Omkar Vinit Joshi >Priority: Blocker > > Not able to run sleep job on latest trunk > bin/yarn jar > ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT-tests.jar > output :- > Picked up _JAVA_OPTIONS: -Djava.awt.headless=true > java.lang.NoClassDefFoundError: junit/framework/TestCase > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:60) > at > org.apache.hadoop.test.MapredTestDriver.(MapredTestDriver.java:54) > at > org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:123) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 20 more > An example program must be given as the first argument. > Valid program names are: -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-879) Fix tests w.r.t o.a.h.y.server.resourcemanager.Application
[
https://issues.apache.org/jira/browse/YARN-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769098#comment-13769098
]
Hadoop QA commented on YARN-879:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603271/YARN-879-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1943//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1943//console
This message is automatically generated.
> Fix tests w.r.t o.a.h.y.server.resourcemanager.Application
> --
>
> Key: YARN-879
> URL: https://issues.apache.org/jira/browse/YARN-879
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.1.0-beta
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: YARN-879.patch, YARN-879-v2.patch, YARN-879-v3.patch,
> YARN-879-v4.patch
>
>
> getResources() will return a list of containers that allocated by RM.
> However, it is now return null directly. The worse thing is: if LOG.debug is
> enabled, then it will definitely cause NPE exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-261) Ability to kill AM attempts
[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261.patch Attaching a patch implementing "application restart" feature. Effectively, it's what is described here: a current attempt is killed and a new one is started. It'll work only if there are attempts left and recovery is possible - exactly the same conditions which are used when deciding whether to start a new attempt after a failure in the previous one. > Ability to kill AM attempts > --- > > Key: YARN-261 > URL: https://issues.apache.org/jira/browse/YARN-261 > Project: Hadoop YARN > Issue Type: New Feature > Components: api >Affects Versions: 2.0.3-alpha >Reporter: Jason Lowe > Attachments: YARN-261.patch > > > It would be nice if clients could ask for an AM attempt to be killed. This > is analogous to the task attempt kill support provided by MapReduce. > This feature would be useful in a scenario where AM retries are enabled, the > AM supports recovery, and a particular AM attempt is stuck. Currently if > this occurs the user's only recourse is to kill the entire application, > requiring them to resubmit a new application and potentially breaking > downstream dependent jobs if it's part of a bigger workflow. Killing the > attempt would allow a new attempt to be started by the RM without killing the > entire application, and if the AM supports recovery it could potentially save > a lot of work. It could also be useful in workflow scenarios where the > failure of the entire application kills the workflow, but the ability to kill > an attempt can keep the workflow going if the subsequent attempt succeeds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-261) Ability to kill AM attempts
[
https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769178#comment-13769178
]
Xuan Gong commented on YARN-261:
One quick comment:
Let us say RMApp is at RUNNING state, and RMAppAttempt is at RUNNING state,
too. When RMAppAttempt receives the RMAppAttemptEventType.RESTART.
According to the patch, the RMAppAttempt will do:
{code}
.addTransition(
RMAppAttemptState.RUNNING, RMAppAttemptState.KILLED,
+ RMAppAttemptEventType.RESTART,
+ new FinalTransition(RMAppAttemptState.KILLED))
{code}
And at the FinalTransition, when the RMAttemptState is KILLED, it will do
{code}
case KILLED:
{
// don't leave the tracking URL pointing to a non-existent AM
appAttempt.setTrackingUrlToRMAppPage();
appEvent =
new RMAppFailedAttemptEvent(applicationId,
RMAppEventType.ATTEMPT_KILLED,
"Application killed by user.");
}
break;
{code}
The RMApp will receive the RMAppFailedAttemptEvent, unfortunately, this
RMAppEventType.ATTEMPT_KILLED is only valid when RMApp is at KILLed state. So,
in this case, there will be exception throwing out.
I am thinking why we can not just find a way to fail the RMAppAttempt(adding
good diagnostics) instead of killing the RMAppAttempt. If we can fail the
RMAppAttempt, in RMApp, it will check the maxAttemptNumber and start a new
Attempt by itself. In that case, we do not need to write the logic about it.
> Ability to kill AM attempts
> ---
>
> Key: YARN-261
> URL: https://issues.apache.org/jira/browse/YARN-261
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api
>Affects Versions: 2.0.3-alpha
>Reporter: Jason Lowe
> Attachments: YARN-261.patch
>
>
> It would be nice if clients could ask for an AM attempt to be killed. This
> is analogous to the task attempt kill support provided by MapReduce.
> This feature would be useful in a scenario where AM retries are enabled, the
> AM supports recovery, and a particular AM attempt is stuck. Currently if
> this occurs the user's only recourse is to kill the entire application,
> requiring them to resubmit a new application and potentially breaking
> downstream dependent jobs if it's part of a bigger workflow. Killing the
> attempt would allow a new attempt to be started by the RM without killing the
> entire application, and if the AM supports recovery it could potentially save
> a lot of work. It could also be useful in workflow scenarios where the
> failure of the entire application kills the workflow, but the ability to kill
> an attempt can keep the workflow going if the subsequent attempt succeeds.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1206) Container logs link is broken on RM web UI after application finished
[ https://issues.apache.org/jira/browse/YARN-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-1206: -- Fix Version/s: (was: 2.1.1-beta) Haven't reproduced this myself on trunk, but will trust Jian. The presumed issue is not happening on 2.1 on though as this patch is absent. > Container logs link is broken on RM web UI after application finished > - > > Key: YARN-1206 > URL: https://issues.apache.org/jira/browse/YARN-1206 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Priority: Blocker > Labels: 2.1.1-beta > > When container is running, its logs link works properly, but after the > application is finished, the link shows 'Container does not exist.' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1206) Container logs link is broken on RM web UI after application finished
[ https://issues.apache.org/jira/browse/YARN-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769192#comment-13769192 ] Zhijie Shen commented on YARN-1206: --- I observed "Container does not exist." for the containers whose Map/Reduce task is completed, but the whole MR application is still running > Container logs link is broken on RM web UI after application finished > - > > Key: YARN-1206 > URL: https://issues.apache.org/jira/browse/YARN-1206 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Priority: Blocker > Labels: 2.1.1-beta > > When container is running, its logs link works properly, but after the > application is finished, the link shows 'Container does not exist.' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-261) Ability to kill AM attempts
[
https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769194#comment-13769194
]
Hadoop QA commented on YARN-261:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603522/YARN-261.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.TestMRJobClient
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.TestUberAM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1944//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1944//console
This message is automatically generated.
> Ability to kill AM attempts
> ---
>
> Key: YARN-261
> URL: https://issues.apache.org/jira/browse/YARN-261
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api
>Affects Versions: 2.0.3-alpha
>Reporter: Jason Lowe
> Attachments: YARN-261--n2.patch, YARN-261.patch
>
>
> It would be nice if clients could ask for an AM attempt to be killed. This
> is analogous to the task attempt kill support provided by MapReduce.
> This feature would be useful in a scenario where AM retries are enabled, the
> AM supports recovery, and a particular AM attempt is stuck. Currently if
> this occurs the user's only recourse is to kill the entire application,
> requiring them to resubmit a new application and potentially breaking
> downstream dependent jobs if it's part of a bigger workflow. Killing the
> attempt would allow a new attempt to be started by the RM without killing the
> entire application, and if the AM supports recovery it could potentially save
> a lot of work. It could also be useful in workflow scenarios where the
> failure of the entire application kills the workflow, but the ability to kill
> an attempt can keep the workflow going if the subsequent attempt succeeds.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-261) Ability to kill AM attempts
[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261--n2.patch Updated the patch to prevent the exception in the log when processing RMAppEventType.ATTEMPT_KILLED. On failing an attempt instead of additional logic - well, the patch itself consists of: 1) CLI modifications 2) adding the API call and lots of boilerplate needed for that 3) replicating AttemptFailedTransaction into a modified version named AppRestartedTransaction (with different diagnostics) 4) modifying state machines. The 4th part is needed exactly for not firing RMAppEventType.ATTEMPT_KILLED. I missed that part in the prev patch. > Ability to kill AM attempts > --- > > Key: YARN-261 > URL: https://issues.apache.org/jira/browse/YARN-261 > Project: Hadoop YARN > Issue Type: New Feature > Components: api >Affects Versions: 2.0.3-alpha >Reporter: Jason Lowe > Attachments: YARN-261--n2.patch, YARN-261.patch > > > It would be nice if clients could ask for an AM attempt to be killed. This > is analogous to the task attempt kill support provided by MapReduce. > This feature would be useful in a scenario where AM retries are enabled, the > AM supports recovery, and a particular AM attempt is stuck. Currently if > this occurs the user's only recourse is to kill the entire application, > requiring them to resubmit a new application and potentially breaking > downstream dependent jobs if it's part of a bigger workflow. Killing the > attempt would allow a new attempt to be started by the RM without killing the > entire application, and if the AM supports recovery it could potentially save > a lot of work. It could also be useful in workflow scenarios where the > failure of the entire application kills the workflow, but the ability to kill > an attempt can keep the workflow going if the subsequent attempt succeeds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1206) Container logs link is broken on RM web UI after application finished
[ https://issues.apache.org/jira/browse/YARN-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769199#comment-13769199 ] Lohit Vijayarenu commented on YARN-1206: We have seen case where upon enabling log aggregation, container link are broken because logs are aggregated to HDFS. If link is not updated to point ot history server, those will be broken links. Broken here is to say that nodemangaer will not be able to display logs (since they are aggregated to hdfs) One way to reproduce this is, run application to completion, then click application link, something like http://hadoop-rm-host:port/cluster/app/application_1379365648572_0001 Then click on 'logs' link next to ApplicationMaster attempt. This will point to page on NM displaying the below message "Failed while trying to construct the redirect url to the log server. Log Server url may not be configured Unknown container. Container either has not started or has already completed or doesn't belong to this node at all." > Container logs link is broken on RM web UI after application finished > - > > Key: YARN-1206 > URL: https://issues.apache.org/jira/browse/YARN-1206 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Priority: Blocker > Labels: 2.1.1-beta > > When container is running, its logs link works properly, but after the > application is finished, the link shows 'Container does not exist.' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-261) Ability to kill AM attempts
[ https://issues.apache.org/jira/browse/YARN-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated YARN-261: - Attachment: YARN-261--n3.patch Fixing Javadoc warning. > Ability to kill AM attempts > --- > > Key: YARN-261 > URL: https://issues.apache.org/jira/browse/YARN-261 > Project: Hadoop YARN > Issue Type: New Feature > Components: api >Affects Versions: 2.0.3-alpha >Reporter: Jason Lowe > Attachments: YARN-261--n2.patch, YARN-261--n3.patch, YARN-261.patch > > > It would be nice if clients could ask for an AM attempt to be killed. This > is analogous to the task attempt kill support provided by MapReduce. > This feature would be useful in a scenario where AM retries are enabled, the > AM supports recovery, and a particular AM attempt is stuck. Currently if > this occurs the user's only recourse is to kill the entire application, > requiring them to resubmit a new application and potentially breaking > downstream dependent jobs if it's part of a bigger workflow. Killing the > attempt would allow a new attempt to be started by the RM without killing the > entire application, and if the AM supports recovery it could potentially save > a lot of work. It could also be useful in workflow scenarios where the > failure of the entire application kills the workflow, but the ability to kill > an attempt can keep the workflow going if the subsequent attempt succeeds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1206) Container logs link is broken on RM web UI after application finished
[ https://issues.apache.org/jira/browse/YARN-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-1206: -- Description: With log aggregation disabled, when container is running, its logs link works properly, but after the application is finished, the link shows 'Container does not exist.' (was: When container is running, its logs link works properly, but after the application is finished, the link shows 'Container does not exist.') > Container logs link is broken on RM web UI after application finished > - > > Key: YARN-1206 > URL: https://issues.apache.org/jira/browse/YARN-1206 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Priority: Blocker > Labels: 2.1.1-beta > > With log aggregation disabled, when container is running, its logs link works > properly, but after the application is finished, the link shows 'Container > does not exist.' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1206) Container logs link is broken on RM web UI after application finished
[ https://issues.apache.org/jira/browse/YARN-1206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769201#comment-13769201 ] Jian He commented on YARN-1206: --- Hi [~lohit], thanks for clarifying, here the situation is that log aggregation is disabled. > Container logs link is broken on RM web UI after application finished > - > > Key: YARN-1206 > URL: https://issues.apache.org/jira/browse/YARN-1206 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Priority: Blocker > Labels: 2.1.1-beta > > With log aggregation disabled, when container is running, its logs link works > properly, but after the application is finished, the link shows 'Container > does not exist.' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1197) Support increasing resources of an allocated container
[ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769233#comment-13769233 ] Bikas Saha commented on YARN-1197: -- Increasing resources for a container while in acquired state is not different from waiting for some more time on the RM and allocating the larger container in the first attempt, right? Also, the RM starts a timer for each acquired container and expects the container to be launched on the NM before the timer expires. So we dont have too much time for the container to be launched and thus we cannot wait for increasing the resources. To be useful, we have to be able to increase the resources of a running container. I agree that its a significant change. So making the change will need a more thorough investigation and clear design proposal. Your help in making this happen is most welcome! > Support increasing resources of an allocated container > -- > > Key: YARN-1197 > URL: https://issues.apache.org/jira/browse/YARN-1197 > Project: Hadoop YARN > Issue Type: Task > Components: api, nodemanager, resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Wangda Tan > > Currently, YARN cannot support merge several containers in one node to a big > container, which can make us incrementally ask resources, merge them to a > bigger one, and launch our processes. The user scenario is described in the > comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1197) Support increasing resources of an allocated container
[
https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769256#comment-13769256
]
Wangda Tan commented on YARN-1197:
--
{quote}
Increasing resources for a container while in acquired state is not different
from waiting for some more time on the RM and allocating the larger container
in the first attempt, right?
{quote}
I think there's a little difference here, because waiting for resource for a
big container in the first attempt, scheduler will put the the request to
"reservedContainer" at FSSchedulableNode or FiCaSchedulerNode. This will be
considered as an exception, RM will try to satisfy such reserved container
first when many different requests existed at the same time in a same node.
But if we try to ask more resource in an acquired container, I don't know
what's your preferring, do you want to create another "exception" which can put
an "acquired container" to *ScheduableNode to make it can get prior proceeded
or just simply make the request as a normal resource request?
{quote}
Also, the RM starts a timer for each acquired container and expects the
container to be launched on the NM before the timer expires. So we dont have
too much time for the container to be launched and thus we cannot wait for
increasing the resources.
{quote}
I don't know if we can refresh(receivePing) the timer for a container when we
successfully increased resource for it?
{quote}
To be useful, we have to be able to increase the resources of a running
container. I agree that its a significant change. So making the change will
need a more thorough investigation and clear design proposal.
{quote}
Agree! I'd like to help moving this forward, I need investigate and consider
end-to-end cases and draft a design proposal for it, once I've some ideas or
question, I will let you know :)
Thanks
> Support increasing resources of an allocated container
> --
>
> Key: YARN-1197
> URL: https://issues.apache.org/jira/browse/YARN-1197
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, nodemanager, resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Wangda Tan
>
> Currently, YARN cannot support merge several containers in one node to a big
> container, which can make us incrementally ask resources, merge them to a
> bigger one, and launch our processes. The user scenario is described in the
> comments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
