[jira] [Commented] (YARN-3030) set up ATS writer with basic request serving structure and lifecycle
[ https://issues.apache.org/jira/browse/YARN-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281243#comment-14281243 ] Zhijie Shen commented on YARN-3030: --- I'm not sure if possible to create own jenkins job. Previously when we developed on branch YARN-321, we usually built the source code and ran through the unit tests locally before committing each patch. Once we merged the branch back to trunk, we did one pass of code cleanup and fixed warnings. > set up ATS writer with basic request serving structure and lifecycle > > > Key: YARN-3030 > URL: https://issues.apache.org/jira/browse/YARN-3030 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: YARN-3030.001.patch > > > Per design in YARN-2928, create an ATS writer as a service, and implement the > basic service structure including the lifecycle management. > Also, as part of this JIRA, we should come up with the ATS client API for > sending requests to this ATS writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3030) set up ATS writer with basic request serving structure and lifecycle
[ https://issues.apache.org/jira/browse/YARN-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281089#comment-14281089 ] Sangjin Lee commented on YARN-3030: --- OK, the regular pre-commit jenkins build is based on the trunk so my incremental patch is not going to work with the regular pre-commit jenkins build. Then, I don't think we should generate cumulative patches (diff from the trunk) as those would be impossible to review. Should we create our own jenkins job? How does this process work with branch development? > set up ATS writer with basic request serving structure and lifecycle > > > Key: YARN-3030 > URL: https://issues.apache.org/jira/browse/YARN-3030 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: YARN-3030.001.patch > > > Per design in YARN-2928, create an ATS writer as a service, and implement the > basic service structure including the lifecycle management. > Also, as part of this JIRA, we should come up with the ATS client API for > sending requests to this ATS writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3030) set up ATS writer with basic request serving structure and lifecycle
[
https://issues.apache.org/jira/browse/YARN-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281081#comment-14281081
]
Hadoop QA commented on YARN-3030:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692874/YARN-3030.001.patch
against trunk revision 43302f6.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6354//console
This message is automatically generated.
> set up ATS writer with basic request serving structure and lifecycle
>
>
> Key: YARN-3030
> URL: https://issues.apache.org/jira/browse/YARN-3030
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: YARN-3030.001.patch
>
>
> Per design in YARN-2928, create an ATS writer as a service, and implement the
> basic service structure including the lifecycle management.
> Also, as part of this JIRA, we should come up with the ATS client API for
> sending requests to this ATS writer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2815) Remove jline from hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281066#comment-14281066 ] Hudson commented on YARN-2815: -- FAILURE: Integrated in Hadoop-trunk-Commit #6881 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6881/]) YARN-2815. Excluded transitive dependency of JLine in hadoop-yarn-server-common. Contributed by Ferdinand Xu. (zjshen: rev 43302f6f44f97d67069eefdda986b6da2933393e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * hadoop-yarn-project/CHANGES.txt > Remove jline from hadoop-yarn-server-common > --- > > Key: YARN-2815 > URL: https://issues.apache.org/jira/browse/YARN-2815 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: 2.7.0 > > Attachments: YARN-2815.patch > > > hadoop-yarn-server-common bundles ancient jline-0.9.94 as transitive > dependency of zookeeper-3.4.6 (it is used in the ZK CLI and not in Hadoop). > Beeline is moving to JLine2 and have to exclude this jline dependency so that > the hadoop classpath can use their own version (there is a newer JLine 2.12 > that contains history search etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3030) set up ATS writer with basic request serving structure and lifecycle
[
https://issues.apache.org/jira/browse/YARN-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281062#comment-14281062
]
Hadoop QA commented on YARN-3030:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692874/YARN-3030.001.patch
against trunk revision 43302f6.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6353//console
This message is automatically generated.
> set up ATS writer with basic request serving structure and lifecycle
>
>
> Key: YARN-3030
> URL: https://issues.apache.org/jira/browse/YARN-3030
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: YARN-3030.001.patch
>
>
> Per design in YARN-2928, create an ATS writer as a service, and implement the
> basic service structure including the lifecycle management.
> Also, as part of this JIRA, we should come up with the ATS client API for
> sending requests to this ATS writer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1680) availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.
[ https://issues.apache.org/jira/browse/YARN-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281058#comment-14281058 ] Craig Welch commented on YARN-1680: --- Thanks for the update, [~airbots], a couple thoughts: I created [YARN-2848] in the hopes that it would help us to build a solution which could share functionality between various items with similar needs, so that the solution we come up with is build with that in mind. That said, I think we will need to build the solutions independently, and there's no need to do them all at the same time. -re Every time, App asks for blacklist addition, we check whether the nodes in addition are in cluster blacklist or not (O(m), m is the nodes in blacklist addition). If so, remove this node from addition. Unfortunately, I don't think that this can be solved with checks during addition and removal - I believe that we will need to keep a persistent picture of all blacklisted nodes for an application regardless of their cluster state because the two can vary independently and changes after a blacklist request may invalidate things (for example, cluster blacklists just before app blacklists, the app blacklist request is discarded, the cluster reinstates but the app still cannot use the node for reasons different from the nodes cluster availability - we will still include that node in headroom incorrectly...). I also think that, as suggested in [YARN-2848], the only approach I see working for all states is one where there is a last-change indicator of some sort active for the cluster in terms of it's node composition which is held by the application and, when it has updated past the application's last calculation for "app cluster resource" (in this case, the one which omits blacklisted nodes), it re-evaluates state to determine a new "app cluster resource" which it then uses (until a reevaluation is required, again). This should enable the application to have accurate headroom information regardless of the timing of changes and allows for the more complex evaluations which may be needed (rack blacklisting, etc) while minimizing the frequency of those evaluations. I don't think it is necessarily required for blacklisting, but it's worth noting that this could include offloading some of the calculation to the application master (via more informational api's / library functions for calculation) to distribute the cost outward. Again, not necessarily for this case, but I wanted to mention it as I think it is an option now or later on. > availableResources sent to applicationMaster in heartbeat should exclude > blacklistedNodes free memory. > -- > > Key: YARN-1680 > URL: https://issues.apache.org/jira/browse/YARN-1680 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.2.0, 2.3.0 > Environment: SuSE 11 SP2 + Hadoop-2.3 >Reporter: Rohith >Assignee: Chen He > Attachments: YARN-1680-WIP.patch, YARN-1680-v2.patch, > YARN-1680-v2.patch, YARN-1680.patch > > > There are 4 NodeManagers with 8GB each.Total cluster capacity is 32GB.Cluster > slow start is set to 1. > Job is running reducer task occupied 29GB of cluster.One NodeManager(NM-4) is > become unstable(3 Map got killed), MRAppMaster blacklisted unstable > NodeManager(NM-4). All reducer task are running in cluster now. > MRAppMaster does not preempt the reducers because for Reducer preemption > calculation, headRoom is considering blacklisted nodes memory. This makes > jobs to hang forever(ResourceManager does not assing any new containers on > blacklisted nodes but returns availableResouce considers cluster free > memory). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2984) Metrics for container's actual memory usage
[ https://issues.apache.org/jira/browse/YARN-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281053#comment-14281053 ] Hudson commented on YARN-2984: -- FAILURE: Integrated in Hadoop-trunk-Commit #6880 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6880/]) YARN-2984. Metrics for container's actual memory usage. (kasha) (kasha: rev 84198564ba6028d51c1fcf9cdcb87f6ae6e08513) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainerMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsCollectorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt > Metrics for container's actual memory usage > --- > > Key: YARN-2984 > URL: https://issues.apache.org/jira/browse/YARN-2984 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-2984-1.patch, yarn-2984-2.patch, yarn-2984-3.patch, > yarn-2984-prelim.patch > > > It would be nice to capture resource usage per container, for a variety of > reasons. This JIRA is to track memory usage. > YARN-2965 tracks the resource usage on the node, and the two implementations > should reuse code as much as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3030) set up ATS writer with basic request serving structure and lifecycle
[ https://issues.apache.org/jira/browse/YARN-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-3030: -- Attachment: YARN-3030.001.patch Posted patch v.1. (1) It sets up the basic skeleton of the aggregator service hierarchy. BaseAggregatorService sets up a core aggregator service. AppLevelAggregatorService is an extension that is instantiated per app. (2) I added a partial implementation of the PerNodeAggregator. It serves as a REST end point, and provides lifecycle methods so per-app in-memory aggregators can be created and destroyed according to the app lifecycle. (3) Added a few unit tests, and cleaned up the pom a little bit. This is still a skeleton. I simply used the existing common APIs from ATS to fill in the arguments and so on. We'll need to add more as our APIs jell. > set up ATS writer with basic request serving structure and lifecycle > > > Key: YARN-3030 > URL: https://issues.apache.org/jira/browse/YARN-3030 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: YARN-3030.001.patch > > > Per design in YARN-2928, create an ATS writer as a service, and implement the > basic service structure including the lifecycle management. > Also, as part of this JIRA, we should come up with the ATS client API for > sending requests to this ATS writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2815) Remove jline from hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281035#comment-14281035 ] Zhijie Shen commented on YARN-2815: --- Thanks for the comment, Karthik! I'll commit the patch. > Remove jline from hadoop-yarn-server-common > --- > > Key: YARN-2815 > URL: https://issues.apache.org/jira/browse/YARN-2815 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: YARN-2815.patch > > > hadoop-yarn-server-common bundles ancient jline-0.9.94 as transitive > dependency of zookeeper-3.4.6 (it is used in the ZK CLI and not in Hadoop). > Beeline is moving to JLine2 and have to exclude this jline dependency so that > the hadoop classpath can use their own version (there is a newer JLine 2.12 > that contains history search etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2984) Metrics for container's actual memory usage
[ https://issues.apache.org/jira/browse/YARN-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281029#comment-14281029 ] Karthik Kambatla commented on YARN-2984: Thanks for the review, Robert. Checking this in. > Metrics for container's actual memory usage > --- > > Key: YARN-2984 > URL: https://issues.apache.org/jira/browse/YARN-2984 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-2984-1.patch, yarn-2984-2.patch, yarn-2984-3.patch, > yarn-2984-prelim.patch > > > It would be nice to capture resource usage per container, for a variety of > reasons. This JIRA is to track memory usage. > YARN-2965 tracks the resource usage on the node, and the two implementations > should reuse code as much as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2815) Remove jline from hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281019#comment-14281019 ] Karthik Kambatla commented on YARN-2815: I would hold off on upgrading to 3.5 at least until the ZK community calls the release GA. https://issues.apache.org/jira/issues/?jql=project%20%3D%20ZOOKEEPER%20AND%20affectedVersion%20in%20(3.5.0%2C%203.5.1) shows a bunch of issues with 3.5. > Remove jline from hadoop-yarn-server-common > --- > > Key: YARN-2815 > URL: https://issues.apache.org/jira/browse/YARN-2815 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: YARN-2815.patch > > > hadoop-yarn-server-common bundles ancient jline-0.9.94 as transitive > dependency of zookeeper-3.4.6 (it is used in the ZK CLI and not in Hadoop). > Beeline is moving to JLine2 and have to exclude this jline dependency so that > the hadoop classpath can use their own version (there is a newer JLine 2.12 > that contains history search etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (YARN-2815) Remove jline from hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280911#comment-14280911 ] Zhijie Shen edited comment on YARN-2815 at 1/16/15 10:30 PM: - The patch looks good to me. ZK 3.5 is said to be still alpha. It may be a bit aggressive to move to ZK 3.5 now. AFAIK, Hive also depends on ZK 3.4.x. [~brocknoland] and [~ka...@cloudera.com], any comments? was (Author: zjshen): The patch looks good to me. ZK 3.5 is said to be still alpha. It may be a bit aggressive to move to ZK 3.5 now. > Remove jline from hadoop-yarn-server-common > --- > > Key: YARN-2815 > URL: https://issues.apache.org/jira/browse/YARN-2815 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: YARN-2815.patch > > > hadoop-yarn-server-common bundles ancient jline-0.9.94 as transitive > dependency of zookeeper-3.4.6 (it is used in the ZK CLI and not in Hadoop). > Beeline is moving to JLine2 and have to exclude this jline dependency so that > the hadoop classpath can use their own version (there is a newer JLine 2.12 > that contains history search etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2815) Remove jline from hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2815: -- Assignee: Ferdinand Xu > Remove jline from hadoop-yarn-server-common > --- > > Key: YARN-2815 > URL: https://issues.apache.org/jira/browse/YARN-2815 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: YARN-2815.patch > > > hadoop-yarn-server-common bundles ancient jline-0.9.94 as transitive > dependency of zookeeper-3.4.6 (it is used in the ZK CLI and not in Hadoop). > Beeline is moving to JLine2 and have to exclude this jline dependency so that > the hadoop classpath can use their own version (there is a newer JLine 2.12 > that contains history search etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2815) Remove jline from hadoop-yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280911#comment-14280911 ] Zhijie Shen commented on YARN-2815: --- The patch looks good to me. ZK 3.5 is said to be still alpha. It may be a bit aggressive to move to ZK 3.5 now. > Remove jline from hadoop-yarn-server-common > --- > > Key: YARN-2815 > URL: https://issues.apache.org/jira/browse/YARN-2815 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: YARN-2815.patch > > > hadoop-yarn-server-common bundles ancient jline-0.9.94 as transitive > dependency of zookeeper-3.4.6 (it is used in the ZK CLI and not in Hadoop). > Beeline is moving to JLine2 and have to exclude this jline dependency so that > the hadoop classpath can use their own version (there is a newer JLine 2.12 > that contains history search etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3015) yarn classpath command should support same options as hadoop classpath.
[
https://issues.apache.org/jira/browse/YARN-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280800#comment-14280800
]
Hadoop QA commented on YARN-3015:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692826/YARN-3015.005.patch
against trunk revision ec4389c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6352//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6352//console
This message is automatically generated.
> yarn classpath command should support same options as hadoop classpath.
> ---
>
> Key: YARN-3015
> URL: https://issues.apache.org/jira/browse/YARN-3015
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scripts
>Reporter: Chris Nauroth
>Assignee: Varun Saxena
>Priority: Minor
> Attachments: YARN-3015.002.patch, YARN-3015.003.patch,
> YARN-3015.004.patch, YARN-3015.005.patch
>
>
> HADOOP-10903 enhanced the {{hadoop classpath}} command to support optional
> expansion of the wildcards and bundling the classpath into a jar file
> containing a manifest with the Class-Path attribute. The other classpath
> commands should do the same for consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3015) yarn classpath command should support same options as hadoop classpath.
[
https://issues.apache.org/jira/browse/YARN-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-3015:
---
Attachment: YARN-3015.005.patch
> yarn classpath command should support same options as hadoop classpath.
> ---
>
> Key: YARN-3015
> URL: https://issues.apache.org/jira/browse/YARN-3015
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scripts
>Reporter: Chris Nauroth
>Assignee: Varun Saxena
>Priority: Minor
> Attachments: YARN-3015.002.patch, YARN-3015.003.patch,
> YARN-3015.004.patch, YARN-3015.005.patch
>
>
> HADOOP-10903 enhanced the {{hadoop classpath}} command to support optional
> expansion of the wildcards and bundling the classpath into a jar file
> containing a manifest with the Class-Path attribute. The other classpath
> commands should do the same for consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3015) yarn classpath command should support same options as hadoop classpath.
[
https://issues.apache.org/jira/browse/YARN-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-3015:
---
Attachment: (was: YARN-3015.001.patch)
> yarn classpath command should support same options as hadoop classpath.
> ---
>
> Key: YARN-3015
> URL: https://issues.apache.org/jira/browse/YARN-3015
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scripts
>Reporter: Chris Nauroth
>Assignee: Varun Saxena
>Priority: Minor
> Attachments: YARN-3015.002.patch, YARN-3015.003.patch,
> YARN-3015.004.patch
>
>
> HADOOP-10903 enhanced the {{hadoop classpath}} command to support optional
> expansion of the wildcards and bundling the classpath into a jar file
> containing a manifest with the Class-Path attribute. The other classpath
> commands should do the same for consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3069) Document missing properties in yarn-default.xml
Ray Chiang created YARN-3069: Summary: Document missing properties in yarn-default.xml Key: YARN-3069 URL: https://issues.apache.org/jira/browse/YARN-3069 Project: Hadoop YARN Issue Type: Bug Reporter: Ray Chiang Assignee: Ray Chiang The following properties are currently not defined in yarn-default.xml. These properties should either be A) documented in yarn-default.xml OR B) listed as an exception (with comments, e.g. for internal use) in the TestYarnConfigurationFields unit test Any comments for any of the properties below are welcome. org.apache.hadoop.yarn.server.sharedcachemanager.RemoteAppChecker org.apache.hadoop.yarn.server.sharedcachemanager.store.InMemorySCMStore security.applicationhistory.protocol.acl yarn.app.container.log.backups yarn.app.container.log.dir yarn.app.container.log.filesize yarn.client.app-submission.poll-interval yarn.client.application-client-protocol.poll-timeout-ms yarn.is.minicluster yarn.log.server.url yarn.minicluster.control-resource-monitoring yarn.minicluster.fixed.ports yarn.minicluster.use-rpc yarn.node-labels.fs-store.retry-policy-spec yarn.node-labels.fs-store.root-dir yarn.node-labels.manager-class yarn.nodemanager.container-executor.os.sched.priority.adjustment yarn.nodemanager.container-monitor.process-tree.class yarn.nodemanager.disk-health-checker.enable yarn.nodemanager.docker-container-executor.image-name yarn.nodemanager.linux-container-executor.cgroups.delete-timeout-ms yarn.nodemanager.linux-container-executor.group yarn.nodemanager.log.deletion-threads-count yarn.nodemanager.user-home-dir yarn.nodemanager.webapp.https.address yarn.nodemanager.webapp.spnego-keytab-file yarn.nodemanager.webapp.spnego-principal yarn.nodemanager.windows-secure-container-executor.group yarn.resourcemanager.configuration.file-system-based-store yarn.resourcemanager.delegation-token-renewer.thread-count yarn.resourcemanager.delegation.key.update-interval yarn.resourcemanager.delegation.token.max-lifetime yarn.resourcemanager.delegation.token.renew-interval yarn.resourcemanager.history-writer.multi-threaded-dispatcher.pool-size yarn.resourcemanager.metrics.runtime.buckets yarn.resourcemanager.nm-tokens.master-key-rolling-interval-secs yarn.resourcemanager.reservation-system.class yarn.resourcemanager.reservation-system.enable yarn.resourcemanager.reservation-system.plan.follower yarn.resourcemanager.reservation-system.planfollower.time-step yarn.resourcemanager.rm.container-allocation.expiry-interval-ms yarn.resourcemanager.webapp.spnego-keytab-file yarn.resourcemanager.webapp.spnego-principal yarn.scheduler.include-port-in-node-name yarn.timeline-service.delegation.key.update-interval yarn.timeline-service.delegation.token.max-lifetime yarn.timeline-service.delegation.token.renew-interval yarn.timeline-service.generic-application-history.enabled yarn.timeline-service.generic-application-history.fs-history-store.compression-type yarn.timeline-service.generic-application-history.fs-history-store.uri yarn.timeline-service.generic-application-history.store-class yarn.timeline-service.http-cross-origin.enabled yarn.tracking.url.generator -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2664) Improve RM webapp to expose info about reservations.
[
https://issues.apache.org/jira/browse/YARN-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280688#comment-14280688
]
Hadoop QA commented on YARN-2664:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692801/YARN-2664.9.patch
against trunk revision 000ca83.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 5
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6351//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6351//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6351//console
This message is automatically generated.
> Improve RM webapp to expose info about reservations.
>
>
> Key: YARN-2664
> URL: https://issues.apache.org/jira/browse/YARN-2664
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Carlo Curino
>Assignee: Matteo Mazzucchelli
> Attachments: PlannerPage_screenshot.pdf, YARN-2664.1.patch,
> YARN-2664.2.patch, YARN-2664.3.patch, YARN-2664.4.patch, YARN-2664.5.patch,
> YARN-2664.6.patch, YARN-2664.7.patch, YARN-2664.8.patch, YARN-2664.9.patch,
> YARN-2664.patch, legal.patch, screenshot_reservation_UI.pdf
>
>
> YARN-1051 provides a new functionality in the RM to ask for reservation on
> resources. Exposing this through the webapp GUI is important.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3037) create HBase cluster backing storage implementation for ATS writes
[ https://issues.apache.org/jira/browse/YARN-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280613#comment-14280613 ] Zhijie Shen commented on YARN-3037: --- Sorry for the late reply. Yes, please go ahead. [~gtCarrera9] previously also did some work for HBase implementation (YARN-2032). Probably you can coordinate somehow. > create HBase cluster backing storage implementation for ATS writes > -- > > Key: YARN-3037 > URL: https://issues.apache.org/jira/browse/YARN-3037 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Vrushali C > > Per design in YARN-2928, create a backing storage implementation for ATS > writes based on a full HBase cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) Application Timeline Server (ATS) next gen: phase 1
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280565#comment-14280565 ] Zhijie Shen commented on YARN-2928: --- bq. As far as I can tell, all server code has the naming org.apache.hadoop.yarn.server.[feature_name]. Oh, may bad. You're right. bq. Wouldn't it be good to stick with that then? How about the following then? +1 LGTM bq. it may become a source of confusion as the new name sounds very similar. Agree, like "mapred" and "mapreduce". Let's make sure in the documentation we tell the users which package should use for which version of timeline server. > Application Timeline Server (ATS) next gen: phase 1 > --- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Vinod Kumar Vavilapalli >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3050) implement new flow-based ATS queries in the new ATS design
[ https://issues.apache.org/jira/browse/YARN-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280557#comment-14280557 ] Vrushali C commented on YARN-3050: -- Hi Varun Could I take this one up as well? Please let me know. thanks Vrushali > implement new flow-based ATS queries in the new ATS design > -- > > Key: YARN-3050 > URL: https://issues.apache.org/jira/browse/YARN-3050 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > > Implement new flow-based ATS queries in the new ATS design. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2664) Improve RM webapp to expose info about reservations.
[ https://issues.apache.org/jira/browse/YARN-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Mazzucchelli updated YARN-2664: -- Attachment: YARN-2664.9.patch > Improve RM webapp to expose info about reservations. > > > Key: YARN-2664 > URL: https://issues.apache.org/jira/browse/YARN-2664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Matteo Mazzucchelli > Attachments: PlannerPage_screenshot.pdf, YARN-2664.1.patch, > YARN-2664.2.patch, YARN-2664.3.patch, YARN-2664.4.patch, YARN-2664.5.patch, > YARN-2664.6.patch, YARN-2664.7.patch, YARN-2664.8.patch, YARN-2664.9.patch, > YARN-2664.patch, legal.patch, screenshot_reservation_UI.pdf > > > YARN-1051 provides a new functionality in the RM to ask for reservation on > resources. Exposing this through the webapp GUI is important. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3037) create HBase cluster backing storage implementation for ATS writes
[ https://issues.apache.org/jira/browse/YARN-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-3037: -- Assignee: Vrushali C (was: Zhijie Shen) > create HBase cluster backing storage implementation for ATS writes > -- > > Key: YARN-3037 > URL: https://issues.apache.org/jira/browse/YARN-3037 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Vrushali C > > Per design in YARN-2928, create a backing storage implementation for ATS > writes based on a full HBase cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) Application Timeline Server (ATS) next gen: phase 1
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280546#comment-14280546
]
Sangjin Lee commented on YARN-2928:
---
As far as I can tell, all server code has the naming
{{org.apache.hadoop.yarn.server.[feature_name\]}}. Wouldn't it be good to stick
with that then? How about the following then?
{noformat}
common: org.apache.hadoop.yarn.timelineservice.*
client: org.apache.hadoop.yarn.client.api.timelineservice.*
server: org.apache.hadoop.yarn.server.timelineservice.*
{noformat}
One thing is the current ATS code in yarn-api is in
{{org.apache.hadoop.yarn.client.api.records.timeline}} so it may become a
source of confusion as the new name sounds very similar. We'll just need to
find a way to differentiate them (via plenty of documentation or renaming
things).
> Application Timeline Server (ATS) next gen: phase 1
> ---
>
> Key: YARN-2928
> URL: https://issues.apache.org/jira/browse/YARN-2928
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf
>
>
> We have the application timeline server implemented in yarn per YARN-1530 and
> YARN-321. Although it is a great feature, we have recognized several critical
> issues and features that need to be addressed.
> This JIRA proposes the design and implementation changes to address those.
> This is phase 1 of this effort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-3031) create backing storage write interface for ATS writers
[ https://issues.apache.org/jira/browse/YARN-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C reassigned YARN-3031: Assignee: Vrushali C (was: Varun Saxena) > create backing storage write interface for ATS writers > -- > > Key: YARN-3031 > URL: https://issues.apache.org/jira/browse/YARN-3031 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Vrushali C > > Per design in YARN-2928, come up with the interface for the ATS writer to > write to various backing storages. The interface should be created to capture > the right level of abstractions so that it will enable all backing storage > implementations to implement it efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3068) shared secret for trust between RM and AM
Jonathan Maron created YARN-3068: Summary: shared secret for trust between RM and AM Key: YARN-3068 URL: https://issues.apache.org/jira/browse/YARN-3068 Project: Hadoop YARN Issue Type: Bug Components: applications, resourcemanager Reporter: Jonathan Maron When exposing a web endpoint for UI and REST, an AM is dependent on the RM as a proxy for incoming interactions. The RM web proxy supports security features such as SSL and SPNEGO. However, those security mechanisms are not supported by the AM, and supporting them directly at the AM would require some complex implementation details and configuration (not to mention that given the proxying relationship they may be considered somewhat redundant). In order to ensure that there is a measure of security (trust) between the RM web proxy and the AM, the following mechanism is suggested: - The AM will create a shared secret and propagate it to the AM during AM launch (e.g. it could be part of the existing credentials). - The web proxy will leverage the shared secret to encrypt an agreed upon text (e.g. the container ID) and an associated expiry time (to mitigate potential request spoofing). - The AM will decrypt the text leveraging the shared secret and, if successful and the expiry time has not been reached, proceed with the request processing (probably appropriate to perform these checks in the existing AmIpFilter or a specific trust filter). Note that this feature is key to supporting interactions between Knox and AM REST resources, since those interactions depend on trusted proxy support the RM can provide (via its current SPNEGO and "doAs" support), allowing AM's to focus on performing their processing based on the established doAs identity (established at the RM and related to the AM via a trusted path). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3003) Provide API for client to retrieve label to node mapping
[ https://issues.apache.org/jira/browse/YARN-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280427#comment-14280427 ] Varun Saxena commented on YARN-3003: Regarding first point, if we do full regex match this would mean iterating over all the entries. We can probably have a trie as an additonal data structure to store labels to nodes mappings for this and support prefix match. But realistically speaking how many node labels can we have ? Can we have thousands of node labels ? Regex match is only useful if we have a large number of node labels. > Provide API for client to retrieve label to node mapping > > > Key: YARN-3003 > URL: https://issues.apache.org/jira/browse/YARN-3003 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Reporter: Ted Yu >Assignee: Varun Saxena > > Currently YarnClient#getNodeToLabels() returns the mapping from NodeId to set > of labels associated with the node. > Client (such as Slider) may be interested in label to node mapping - given > label, return the nodes with this label. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2861) Timeline DT secret manager should not reuse the RM's configs.
[
https://issues.apache.org/jira/browse/YARN-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280415#comment-14280415
]
Hudson commented on YARN-2861:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2026 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2026/])
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs.
Contributed by Zhijie Shen (jianhe: rev
9e33116d1d8944a393937337b3963e192b9c74d1)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMSecretManagerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* hadoop-yarn-project/CHANGES.txt
> Timeline DT secret manager should not reuse the RM's configs.
> -
>
> Key: YARN-2861
> URL: https://issues.apache.org/jira/browse/YARN-2861
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.7.0
>
> Attachments: YARN-2861.1.patch, YARN-2861.2.patch
>
>
> This is the configs for RM DT secret manager. We should create separate ones
> for timeline DT only.
> {code}
> @Override
> protected void serviceInit(Configuration conf) throws Exception {
> long secretKeyInterval =
> conf.getLong(YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
> long tokenMaxLifetime =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_KEY,
> YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
> long tokenRenewInterval =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
> secretManager = new
> TimelineDelegationTokenSecretManager(secretKeyInterval,
> tokenMaxLifetime, tokenRenewInterval,
> 360);
> secretManager.startThreads();
> serviceAddr = TimelineUtils.getTimelineTokenServiceAddress(getConfig());
> super.init(conf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3064) TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout
[
https://issues.apache.org/jira/browse/YARN-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280414#comment-14280414
]
Hudson commented on YARN-3064:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2026 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2026/])
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync
failure with allocation timeout. (Contributed by Jian He) (junping_du: rev
5d1cca34fa14706586f8bbe7f2a2264757fdcc11)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
> TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with
> allocation timeout
> --
>
> Key: YARN-3064
> URL: https://issues.apache.org/jira/browse/YARN-3064
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Jian He
>Priority: Critical
> Fix For: 3.0.0, 2.7.0
>
> Attachments: YARN-3064.1.patch, YARN-3064.2.patch
>
>
> Noticed consistent tests failure, see:
> https://builds.apache.org/job/PreCommit-YARN-Build/6332//testReport/
> Logs like:
> {code}
> Error Message
> Attempt state is not correct (timedout) expected: but
> was:
> Stacktrace
> java.lang.AssertionError: Attempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1794)
> {code}
> I can reproduce it in local environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3003) Provide API for client to retrieve label to node mapping
[
https://issues.apache.org/jira/browse/YARN-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280403#comment-14280403
]
Varun Saxena commented on YARN-3003:
[~Naganarasimha]
Regarding your second point, I do not think memory footprint of storing NodeId
in NodeLabel class will be too much. We already have labels to NodeLabel
mapping anyways. Let us assume we have 100 node labels in all(even that is on
the higher side I guess) and 4000 nodes. Even if each node label is attached to
every node we will have a set of 4000 NodeIds' for every Node Label. NodeId
merely stores a host and port(let us assume host is of length 20 bytes and port
4 bytes). As string normally allocates more memory than required, let us assume
each NodeId will occupy 50 bytes. This makes it additional memory of
100*4000*50 bytes or around 20 MB. Even if we double it, this wont go upto more
than 40-50 MB of additional memory.
Now if we do not store Labels to Nodes mapping separately, we will have to
iterate over all the 4000 nodes(nodeCollections which is a ConcurrentHashMap).
I do not think thats worth it performance wise, if we consider that additional
memory is just a few MB. I am not sure how often the client will call
{{getLabelsToNodes}} but as this is a public API we cant restrict how client
will behave. If the memory footprint was very high, it would have been a
different matter.
[~leftnoteasy] and [~tedyu] can comment on this as well.
> Provide API for client to retrieve label to node mapping
>
>
> Key: YARN-3003
> URL: https://issues.apache.org/jira/browse/YARN-3003
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client, resourcemanager
>Reporter: Ted Yu
>Assignee: Varun Saxena
>
> Currently YarnClient#getNodeToLabels() returns the mapping from NodeId to set
> of labels associated with the node.
> Client (such as Slider) may be interested in label to node mapping - given
> label, return the nodes with this label.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3003) Provide API for client to retrieve label to node mapping
[
https://issues.apache.org/jira/browse/YARN-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280396#comment-14280396
]
Varun Saxena commented on YARN-3003:
[~Naganarasimha Garla],
Regarding your second point, I do not think memory footprint of storing NodeId
in NodeLabel class will be too much. We already have labels to NodeLabel
mapping anyways. Let us assume we have 100 node labels in all(even that is on
the higher side I guess) and 4000 nodes. Even if each node label is attached to
every node we will have a set of 4000 NodeIds' in every for every Node Label.
NodeId merely stores a host and port(let us assume host is of length 20 bytes
and port 4 bytes). As string normally allocates more memory than required, let
us assume each NodeId will occupy 50 bytes. This makes it additional memory of
100*4000*50 bytes or around 20 MB. Even if we double it, this wont go upto more
than 40-50 MB of additional memory.
Now if we do not store Labels to Nodes mapping separately, we will have to
iterate over all the 4000 nodes(nodeCollections which is a ConcurrentHashMap).
I do not think thats worth it performance wise, if we consider that additional
memory is just a few MB. I am not sure how often the client will call
{{getLabelsToNodes}} but as this is a public API we cant restrict how client
will behave. If the memory footprint was very high, it would have been a
different matter.
[~leftnoteasy] and [~tedyu] can comment on this as well.
> Provide API for client to retrieve label to node mapping
>
>
> Key: YARN-3003
> URL: https://issues.apache.org/jira/browse/YARN-3003
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client, resourcemanager
>Reporter: Ted Yu
>Assignee: Varun Saxena
>
> Currently YarnClient#getNodeToLabels() returns the mapping from NodeId to set
> of labels associated with the node.
> Client (such as Slider) may be interested in label to node mapping - given
> label, return the nodes with this label.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3064) TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout
[
https://issues.apache.org/jira/browse/YARN-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280353#comment-14280353
]
Hudson commented on YARN-3064:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #76 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/76/])
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync
failure with allocation timeout. (Contributed by Jian He) (junping_du: rev
5d1cca34fa14706586f8bbe7f2a2264757fdcc11)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
> TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with
> allocation timeout
> --
>
> Key: YARN-3064
> URL: https://issues.apache.org/jira/browse/YARN-3064
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Jian He
>Priority: Critical
> Fix For: 3.0.0, 2.7.0
>
> Attachments: YARN-3064.1.patch, YARN-3064.2.patch
>
>
> Noticed consistent tests failure, see:
> https://builds.apache.org/job/PreCommit-YARN-Build/6332//testReport/
> Logs like:
> {code}
> Error Message
> Attempt state is not correct (timedout) expected: but
> was:
> Stacktrace
> java.lang.AssertionError: Attempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1794)
> {code}
> I can reproduce it in local environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2861) Timeline DT secret manager should not reuse the RM's configs.
[
https://issues.apache.org/jira/browse/YARN-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280354#comment-14280354
]
Hudson commented on YARN-2861:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #76 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/76/])
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs.
Contributed by Zhijie Shen (jianhe: rev
9e33116d1d8944a393937337b3963e192b9c74d1)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMSecretManagerService.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
> Timeline DT secret manager should not reuse the RM's configs.
> -
>
> Key: YARN-2861
> URL: https://issues.apache.org/jira/browse/YARN-2861
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.7.0
>
> Attachments: YARN-2861.1.patch, YARN-2861.2.patch
>
>
> This is the configs for RM DT secret manager. We should create separate ones
> for timeline DT only.
> {code}
> @Override
> protected void serviceInit(Configuration conf) throws Exception {
> long secretKeyInterval =
> conf.getLong(YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
> long tokenMaxLifetime =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_KEY,
> YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
> long tokenRenewInterval =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
> secretManager = new
> TimelineDelegationTokenSecretManager(secretKeyInterval,
> tokenMaxLifetime, tokenRenewInterval,
> 360);
> secretManager.startThreads();
> serviceAddr = TimelineUtils.getTimelineTokenServiceAddress(getConfig());
> super.init(conf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3003) Provide API for client to retrieve label to node mapping
[ https://issues.apache.org/jira/browse/YARN-3003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280315#comment-14280315 ] Naganarasimha G R commented on YARN-3003: - Hi [~leftnoteasy] & [~varun_saxena] There are 2 concerns : # If the intention of the using this interface by the clients is to form the label expression for the application submission, then better to support this interface to accept a label expression and return the nodes matching the node labels expression itself right? may be in future users might require an approach to get nodes matching a label expression to configure for a queue or an application, so i feel better to support for expression itself. Please provide your opinion # If they want to only retrieve the label-> nodes mapping, then is it good to maintain all the node information in the NodeLabel class and ensure consistency on modifications ? This data kept in NodeLabel class will be used only by this interface and only If the client is repeatedly using this interface then it makes sense(good performance) to store this mapping if not it will be just lying in memory > Provide API for client to retrieve label to node mapping > > > Key: YARN-3003 > URL: https://issues.apache.org/jira/browse/YARN-3003 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, resourcemanager >Reporter: Ted Yu >Assignee: Varun Saxena > > Currently YarnClient#getNodeToLabels() returns the mapping from NodeId to set > of labels associated with the node. > Client (such as Slider) may be interested in label to node mapping - given > label, return the nodes with this label. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3005) [JDK7] Use switch statement for String instead of if-else statement in RegistrySecurity.java
[
https://issues.apache.org/jira/browse/YARN-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280314#comment-14280314
]
Hudson commented on YARN-3005:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/])
YARN-3005. [JDK7] Use switch statement for String instead of if-else statement
in RegistrySecurity.java (Contributed by Kengo Seki) (aajisaka: rev
533e551eb42af188535aeb0ab35f8ebf150a0da1)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java
* hadoop-yarn-project/CHANGES.txt
> [JDK7] Use switch statement for String instead of if-else statement in
> RegistrySecurity.java
>
>
> Key: YARN-3005
> URL: https://issues.apache.org/jira/browse/YARN-3005
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Akira AJISAKA
>Priority: Trivial
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: YARN-3005.001.patch, YARN-3005.002.patch
>
>
> Since we have moved to JDK7, we can refactor the below if-else statement for
> String.
> {code}
> // TODO JDK7 SWITCH
> if (REGISTRY_CLIENT_AUTH_KERBEROS.equals(auth)) {
> access = AccessPolicy.sasl;
> } else if (REGISTRY_CLIENT_AUTH_DIGEST.equals(auth)) {
> access = AccessPolicy.digest;
> } else if (REGISTRY_CLIENT_AUTH_ANONYMOUS.equals(auth)) {
> access = AccessPolicy.anon;
> } else {
> throw new ServiceStateException(E_UNKNOWN_AUTHENTICATION_MECHANISM
> + "\"" + auth + "\"");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2861) Timeline DT secret manager should not reuse the RM's configs.
[
https://issues.apache.org/jira/browse/YARN-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280308#comment-14280308
]
Hudson commented on YARN-2861:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/])
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs.
Contributed by Zhijie Shen (jianhe: rev
9e33116d1d8944a393937337b3963e192b9c74d1)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMSecretManagerService.java
> Timeline DT secret manager should not reuse the RM's configs.
> -
>
> Key: YARN-2861
> URL: https://issues.apache.org/jira/browse/YARN-2861
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.7.0
>
> Attachments: YARN-2861.1.patch, YARN-2861.2.patch
>
>
> This is the configs for RM DT secret manager. We should create separate ones
> for timeline DT only.
> {code}
> @Override
> protected void serviceInit(Configuration conf) throws Exception {
> long secretKeyInterval =
> conf.getLong(YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
> long tokenMaxLifetime =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_KEY,
> YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
> long tokenRenewInterval =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
> secretManager = new
> TimelineDelegationTokenSecretManager(secretKeyInterval,
> tokenMaxLifetime, tokenRenewInterval,
> 360);
> secretManager.startThreads();
> serviceAddr = TimelineUtils.getTimelineTokenServiceAddress(getConfig());
> super.init(conf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3064) TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout
[
https://issues.apache.org/jira/browse/YARN-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280307#comment-14280307
]
Hudson commented on YARN-3064:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/])
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync
failure with allocation timeout. (Contributed by Jian He) (junping_du: rev
5d1cca34fa14706586f8bbe7f2a2264757fdcc11)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
* hadoop-yarn-project/CHANGES.txt
> TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with
> allocation timeout
> --
>
> Key: YARN-3064
> URL: https://issues.apache.org/jira/browse/YARN-3064
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Jian He
>Priority: Critical
> Fix For: 3.0.0, 2.7.0
>
> Attachments: YARN-3064.1.patch, YARN-3064.2.patch
>
>
> Noticed consistent tests failure, see:
> https://builds.apache.org/job/PreCommit-YARN-Build/6332//testReport/
> Logs like:
> {code}
> Error Message
> Attempt state is not correct (timedout) expected: but
> was:
> Stacktrace
> java.lang.AssertionError: Attempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1794)
> {code}
> I can reproduce it in local environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3005) [JDK7] Use switch statement for String instead of if-else statement in RegistrySecurity.java
[
https://issues.apache.org/jira/browse/YARN-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280293#comment-14280293
]
Hudson commented on YARN-3005:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/])
YARN-3005. [JDK7] Use switch statement for String instead of if-else statement
in RegistrySecurity.java (Contributed by Kengo Seki) (aajisaka: rev
533e551eb42af188535aeb0ab35f8ebf150a0da1)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java
> [JDK7] Use switch statement for String instead of if-else statement in
> RegistrySecurity.java
>
>
> Key: YARN-3005
> URL: https://issues.apache.org/jira/browse/YARN-3005
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Akira AJISAKA
>Priority: Trivial
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: YARN-3005.001.patch, YARN-3005.002.patch
>
>
> Since we have moved to JDK7, we can refactor the below if-else statement for
> String.
> {code}
> // TODO JDK7 SWITCH
> if (REGISTRY_CLIENT_AUTH_KERBEROS.equals(auth)) {
> access = AccessPolicy.sasl;
> } else if (REGISTRY_CLIENT_AUTH_DIGEST.equals(auth)) {
> access = AccessPolicy.digest;
> } else if (REGISTRY_CLIENT_AUTH_ANONYMOUS.equals(auth)) {
> access = AccessPolicy.anon;
> } else {
> throw new ServiceStateException(E_UNKNOWN_AUTHENTICATION_MECHANISM
> + "\"" + auth + "\"");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2861) Timeline DT secret manager should not reuse the RM's configs.
[
https://issues.apache.org/jira/browse/YARN-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280287#comment-14280287
]
Hudson commented on YARN-2861:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/])
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs.
Contributed by Zhijie Shen (jianhe: rev
9e33116d1d8944a393937337b3963e192b9c74d1)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMSecretManagerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* hadoop-yarn-project/CHANGES.txt
> Timeline DT secret manager should not reuse the RM's configs.
> -
>
> Key: YARN-2861
> URL: https://issues.apache.org/jira/browse/YARN-2861
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.7.0
>
> Attachments: YARN-2861.1.patch, YARN-2861.2.patch
>
>
> This is the configs for RM DT secret manager. We should create separate ones
> for timeline DT only.
> {code}
> @Override
> protected void serviceInit(Configuration conf) throws Exception {
> long secretKeyInterval =
> conf.getLong(YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
> long tokenMaxLifetime =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_KEY,
> YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
> long tokenRenewInterval =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
> secretManager = new
> TimelineDelegationTokenSecretManager(secretKeyInterval,
> tokenMaxLifetime, tokenRenewInterval,
> 360);
> secretManager.startThreads();
> serviceAddr = TimelineUtils.getTimelineTokenServiceAddress(getConfig());
> super.init(conf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3064) TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout
[
https://issues.apache.org/jira/browse/YARN-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280286#comment-14280286
]
Hudson commented on YARN-3064:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/])
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync
failure with allocation timeout. (Contributed by Jian He) (junping_du: rev
5d1cca34fa14706586f8bbe7f2a2264757fdcc11)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
> TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with
> allocation timeout
> --
>
> Key: YARN-3064
> URL: https://issues.apache.org/jira/browse/YARN-3064
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Jian He
>Priority: Critical
> Fix For: 3.0.0, 2.7.0
>
> Attachments: YARN-3064.1.patch, YARN-3064.2.patch
>
>
> Noticed consistent tests failure, see:
> https://builds.apache.org/job/PreCommit-YARN-Build/6332//testReport/
> Logs like:
> {code}
> Error Message
> Attempt state is not correct (timedout) expected: but
> was:
> Stacktrace
> java.lang.AssertionError: Attempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1794)
> {code}
> I can reproduce it in local environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2990) FairScheduler's delay-scheduling always waits for node-local and rack-local delays, even for off-rack-only requests
[
https://issues.apache.org/jira/browse/YARN-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280274#comment-14280274
]
Hadoop QA commented on YARN-2990:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692749/yarn-2990-0.patch
against trunk revision 000ca83.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6350//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6350//console
This message is automatically generated.
> FairScheduler's delay-scheduling always waits for node-local and rack-local
> delays, even for off-rack-only requests
> ---
>
> Key: YARN-2990
> URL: https://issues.apache.org/jira/browse/YARN-2990
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.6.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: yarn-2990-0.patch, yarn-2990-test.patch
>
>
> Looking at the FairScheduler, it appears the node/rack locality delays are
> used for all requests, even those that are only off-rack.
> More details in comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2990) FairScheduler's delay-scheduling always waits for node-local and rack-local delays, even for off-rack-only requests
[ https://issues.apache.org/jira/browse/YARN-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280238#comment-14280238 ] Karthik Kambatla commented on YARN-2990: Handling rack-local requests the same way could be much more involved than handling off-switch requests. ResourceRequest has hostName, the RM/scheduler has mapping of NodeId -> SchedulerNode. Checking whether the hostName corresponds to a node or rack can't be achieved through a simple map.containsKey() and requires iterating through all racks/nodes which can be quite expensive. How about we handle off-switch requests here and work on the rack-local requests in a follow-up JIRA? This lets us take care of the AM container which is very common. > FairScheduler's delay-scheduling always waits for node-local and rack-local > delays, even for off-rack-only requests > --- > > Key: YARN-2990 > URL: https://issues.apache.org/jira/browse/YARN-2990 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-2990-0.patch, yarn-2990-test.patch > > > Looking at the FairScheduler, it appears the node/rack locality delays are > used for all requests, even those that are only off-rack. > More details in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3067) REST API for System Configuration
Doug Haigh created YARN-3067: Summary: REST API for System Configuration Key: YARN-3067 URL: https://issues.apache.org/jira/browse/YARN-3067 Project: Hadoop YARN Issue Type: Improvement Components: api, client Affects Versions: 2.6.0 Environment: REST API Reporter: Doug Haigh While the name node exposes its configuration via :/conf and resource manager exposes its configuration via :/conf, neither provide a complete picture of the system's configuration for applications that use the YARN REST API. This JIRA is to request a REST API to get all services' configuration information whether it is stored in the resource manager or something external like zookeeper. Essentially this would return information similar to what /conf and /conf return, but be guaranteed to have all information so that a REST API user would not require Hadoop configuration files to know how services are setup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3009) TimelineWebServices always parses primary and secondary filters as numbers if first char is a number
[
https://issues.apache.org/jira/browse/YARN-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280191#comment-14280191
]
Naganarasimha G R commented on YARN-3009:
-
Hi [~zjshen],
I had tested exponential numeric value {{
assertTrue(Number.class.isAssignableFrom(TimelineWebServices.readValue("" +
Float.MAX_VALUE).getClass()));}}
but as you informed the work arnd solution will not work in few cases like
{{123.45E+6}} as the representation of double is diff {{123.45E+6 => 1.2345E+8
}}. So can we close this issue as will not fix ?
> TimelineWebServices always parses primary and secondary filters as numbers if
> first char is a number
>
>
> Key: YARN-3009
> URL: https://issues.apache.org/jira/browse/YARN-3009
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Chris K Wensel
>Assignee: Naganarasimha G R
> Attachments: YARN-3009.20150108-1.patch, YARN-3009.20150111-1.patch
>
>
> If you pass a filter value that starts with a number (7CCA...), the filter
> value will be parsed into the Number '7' causing the filter to fail the
> search.
> Should be noted the actual value as stored via a PUT operation is properly
> parsed and stored as a String.
> This manifests as a very hard to identify issue with DAGClient in Apache Tez
> and naming dags/vertices with alphanumeric guid values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2990) FairScheduler's delay-scheduling always waits for node-local and rack-local delays, even for off-rack-only requests
[ https://issues.apache.org/jira/browse/YARN-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280176#comment-14280176 ] Karthik Kambatla commented on YARN-2990: Discussed this with Sandy offline. We agreed on the following approach when assigning work from an app to a node: # Consider the app's node-local requests for that node. # If there are no node-local requests for the app (for any node in the cluster), or if the allowed locality level is rack-local or off-switch, consider the app's rack-local requests for that rack. # If there are no node-local or rack-local requests (for any node or rack), or if the allowed locality level is off-switch, consider the app's off-switch requests. Attached is a patch (v0) that works for off-switch requests. I tested the patch both using the unit test and on a cluster, and there is no longer a delay to launch the AM container. To handle the rack-local requests, we should be able to differentiate between node and rack names in the data-structure that holds all the ResourceRequests. Or, we could keep counters when adding RRs and assigning containers. > FairScheduler's delay-scheduling always waits for node-local and rack-local > delays, even for off-rack-only requests > --- > > Key: YARN-2990 > URL: https://issues.apache.org/jira/browse/YARN-2990 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-2990-0.patch, yarn-2990-test.patch > > > Looking at the FairScheduler, it appears the node/rack locality delays are > used for all requests, even those that are only off-rack. > More details in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2990) FairScheduler's delay-scheduling always waits for node-local and rack-local delays, even for off-rack-only requests
[ https://issues.apache.org/jira/browse/YARN-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-2990: --- Attachment: yarn-2990-0.patch > FairScheduler's delay-scheduling always waits for node-local and rack-local > delays, even for off-rack-only requests > --- > > Key: YARN-2990 > URL: https://issues.apache.org/jira/browse/YARN-2990 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: yarn-2990-0.patch, yarn-2990-test.patch > > > Looking at the FairScheduler, it appears the node/rack locality delays are > used for all requests, even those that are only off-rack. > More details in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2861) Timeline DT secret manager should not reuse the RM's configs.
[
https://issues.apache.org/jira/browse/YARN-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280098#comment-14280098
]
Hudson commented on YARN-2861:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/809/])
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs.
Contributed by Zhijie Shen (jianhe: rev
9e33116d1d8944a393937337b3963e192b9c74d1)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMSecretManagerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java
> Timeline DT secret manager should not reuse the RM's configs.
> -
>
> Key: YARN-2861
> URL: https://issues.apache.org/jira/browse/YARN-2861
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.7.0
>
> Attachments: YARN-2861.1.patch, YARN-2861.2.patch
>
>
> This is the configs for RM DT secret manager. We should create separate ones
> for timeline DT only.
> {code}
> @Override
> protected void serviceInit(Configuration conf) throws Exception {
> long secretKeyInterval =
> conf.getLong(YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
> long tokenMaxLifetime =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_KEY,
> YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
> long tokenRenewInterval =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
> secretManager = new
> TimelineDelegationTokenSecretManager(secretKeyInterval,
> tokenMaxLifetime, tokenRenewInterval,
> 360);
> secretManager.startThreads();
> serviceAddr = TimelineUtils.getTimelineTokenServiceAddress(getConfig());
> super.init(conf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3005) [JDK7] Use switch statement for String instead of if-else statement in RegistrySecurity.java
[
https://issues.apache.org/jira/browse/YARN-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280104#comment-14280104
]
Hudson commented on YARN-3005:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/809/])
YARN-3005. [JDK7] Use switch statement for String instead of if-else statement
in RegistrySecurity.java (Contributed by Kengo Seki) (aajisaka: rev
533e551eb42af188535aeb0ab35f8ebf150a0da1)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java
* hadoop-yarn-project/CHANGES.txt
> [JDK7] Use switch statement for String instead of if-else statement in
> RegistrySecurity.java
>
>
> Key: YARN-3005
> URL: https://issues.apache.org/jira/browse/YARN-3005
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Akira AJISAKA
>Priority: Trivial
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: YARN-3005.001.patch, YARN-3005.002.patch
>
>
> Since we have moved to JDK7, we can refactor the below if-else statement for
> String.
> {code}
> // TODO JDK7 SWITCH
> if (REGISTRY_CLIENT_AUTH_KERBEROS.equals(auth)) {
> access = AccessPolicy.sasl;
> } else if (REGISTRY_CLIENT_AUTH_DIGEST.equals(auth)) {
> access = AccessPolicy.digest;
> } else if (REGISTRY_CLIENT_AUTH_ANONYMOUS.equals(auth)) {
> access = AccessPolicy.anon;
> } else {
> throw new ServiceStateException(E_UNKNOWN_AUTHENTICATION_MECHANISM
> + "\"" + auth + "\"");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3064) TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout
[
https://issues.apache.org/jira/browse/YARN-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280097#comment-14280097
]
Hudson commented on YARN-3064:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/809/])
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync
failure with allocation timeout. (Contributed by Jian He) (junping_du: rev
5d1cca34fa14706586f8bbe7f2a2264757fdcc11)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
> TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with
> allocation timeout
> --
>
> Key: YARN-3064
> URL: https://issues.apache.org/jira/browse/YARN-3064
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Jian He
>Priority: Critical
> Fix For: 3.0.0, 2.7.0
>
> Attachments: YARN-3064.1.patch, YARN-3064.2.patch
>
>
> Noticed consistent tests failure, see:
> https://builds.apache.org/job/PreCommit-YARN-Build/6332//testReport/
> Logs like:
> {code}
> Error Message
> Attempt state is not correct (timedout) expected: but
> was:
> Stacktrace
> java.lang.AssertionError: Attempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1794)
> {code}
> I can reproduce it in local environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3005) [JDK7] Use switch statement for String instead of if-else statement in RegistrySecurity.java
[
https://issues.apache.org/jira/browse/YARN-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280090#comment-14280090
]
Hudson commented on YARN-3005:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/])
YARN-3005. [JDK7] Use switch statement for String instead of if-else statement
in RegistrySecurity.java (Contributed by Kengo Seki) (aajisaka: rev
533e551eb42af188535aeb0ab35f8ebf150a0da1)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java
> [JDK7] Use switch statement for String instead of if-else statement in
> RegistrySecurity.java
>
>
> Key: YARN-3005
> URL: https://issues.apache.org/jira/browse/YARN-3005
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 2.7.0
>Reporter: Akira AJISAKA
>Priority: Trivial
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: YARN-3005.001.patch, YARN-3005.002.patch
>
>
> Since we have moved to JDK7, we can refactor the below if-else statement for
> String.
> {code}
> // TODO JDK7 SWITCH
> if (REGISTRY_CLIENT_AUTH_KERBEROS.equals(auth)) {
> access = AccessPolicy.sasl;
> } else if (REGISTRY_CLIENT_AUTH_DIGEST.equals(auth)) {
> access = AccessPolicy.digest;
> } else if (REGISTRY_CLIENT_AUTH_ANONYMOUS.equals(auth)) {
> access = AccessPolicy.anon;
> } else {
> throw new ServiceStateException(E_UNKNOWN_AUTHENTICATION_MECHANISM
> + "\"" + auth + "\"");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3064) TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout
[
https://issues.apache.org/jira/browse/YARN-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280083#comment-14280083
]
Hudson commented on YARN-3064:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/])
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync
failure with allocation timeout. (Contributed by Jian He) (junping_du: rev
5d1cca34fa14706586f8bbe7f2a2264757fdcc11)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
> TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with
> allocation timeout
> --
>
> Key: YARN-3064
> URL: https://issues.apache.org/jira/browse/YARN-3064
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Jian He
>Priority: Critical
> Fix For: 3.0.0, 2.7.0
>
> Attachments: YARN-3064.1.patch, YARN-3064.2.patch
>
>
> Noticed consistent tests failure, see:
> https://builds.apache.org/job/PreCommit-YARN-Build/6332//testReport/
> Logs like:
> {code}
> Error Message
> Attempt state is not correct (timedout) expected: but
> was:
> Stacktrace
> java.lang.AssertionError: Attempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1794)
> {code}
> I can reproduce it in local environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2861) Timeline DT secret manager should not reuse the RM's configs.
[
https://issues.apache.org/jira/browse/YARN-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280084#comment-14280084
]
Hudson commented on YARN-2861:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/])
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs.
Contributed by Zhijie Shen (jianhe: rev
9e33116d1d8944a393937337b3963e192b9c74d1)
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMSecretManagerService.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineDelegationTokenSecretManagerService.java
> Timeline DT secret manager should not reuse the RM's configs.
> -
>
> Key: YARN-2861
> URL: https://issues.apache.org/jira/browse/YARN-2861
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Fix For: 2.7.0
>
> Attachments: YARN-2861.1.patch, YARN-2861.2.patch
>
>
> This is the configs for RM DT secret manager. We should create separate ones
> for timeline DT only.
> {code}
> @Override
> protected void serviceInit(Configuration conf) throws Exception {
> long secretKeyInterval =
> conf.getLong(YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
> long tokenMaxLifetime =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_KEY,
> YarnConfiguration.DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
> long tokenRenewInterval =
> conf.getLong(YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
> YarnConfiguration.DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
> secretManager = new
> TimelineDelegationTokenSecretManager(secretKeyInterval,
> tokenMaxLifetime, tokenRenewInterval,
> 360);
> secretManager.startThreads();
> serviceAddr = TimelineUtils.getTimelineTokenServiceAddress(getConfig());
> super.init(conf);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3064) TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout
[
https://issues.apache.org/jira/browse/YARN-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279945#comment-14279945
]
Hudson commented on YARN-3064:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6875 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6875/])
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync
failure with allocation timeout. (Contributed by Jian He) (junping_du: rev
5d1cca34fa14706586f8bbe7f2a2264757fdcc11)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
* hadoop-yarn-project/CHANGES.txt
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestContainerResourceUsage.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java
> TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with
> allocation timeout
> --
>
> Key: YARN-3064
> URL: https://issues.apache.org/jira/browse/YARN-3064
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Jian He
>Priority: Critical
> Fix For: 3.0.0, 2.7.0
>
> Attachments: YARN-3064.1.patch, YARN-3064.2.patch
>
>
> Noticed consistent tests failure, see:
> https://builds.apache.org/job/PreCommit-YARN-Build/6332//testReport/
> Logs like:
> {code}
> Error Message
> Attempt state is not correct (timedout) expected: but
> was:
> Stacktrace
> java.lang.AssertionError: Attempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1794)
> {code}
> I can reproduce it in local environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3064) TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout
[
https://issues.apache.org/jira/browse/YARN-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated YARN-3064:
-
Summary: TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync
failure with allocation timeout (was:
TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with
allocation timeout in trunk)
> TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with
> allocation timeout
> --
>
> Key: YARN-3064
> URL: https://issues.apache.org/jira/browse/YARN-3064
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Reporter: Wangda Tan
>Assignee: Jian He
>Priority: Critical
> Fix For: 3.0.0, 2.7.0
>
> Attachments: YARN-3064.1.patch, YARN-3064.2.patch
>
>
> Noticed consistent tests failure, see:
> https://builds.apache.org/job/PreCommit-YARN-Build/6332//testReport/
> Logs like:
> {code}
> Error Message
> Attempt state is not correct (timedout) expected: but
> was:
> Stacktrace
> java.lang.AssertionError: Attempt state is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:152)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1794)
> {code}
> I can reproduce it in local environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
