[jira] [Commented] (YARN-3102) Decommisioned Nodes not listed in Web UI
[ https://issues.apache.org/jira/browse/YARN-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298259#comment-14298259 ] Bibin A Chundatt commented on YARN-3102: Thanks Devaraj K for checking the issue All the time when i start the server first 2 nodemanagers and then Resource manager. As mentioned in defect describtion when i start Resourcemanager exclude file was configured with NM1 host name and both Node Manager 1 and 2 were up . Then resource mangers starts up is does send signal also 2015-01-30 17:15:11,549 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Disallowed NodeManager from HOST-10-19-92-128, Sending SHUTDOWN signal to the NodeManager. But table is shown as empty > Decommisioned Nodes not listed in Web UI > > > Key: YARN-3102 > URL: https://issues.apache.org/jira/browse/YARN-3102 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 > Environment: 2 Node Manager and 1 Resource Manager >Reporter: Bibin A Chundatt >Assignee: Naganarasimha G R >Priority: Minor > > Configure yarn.resourcemanager.nodes.exclude-path in yarn-site.xml to > yarn.exlude file In RM1 machine > Add Yarn.exclude with NM1 Host Name > Start the node as listed below NM1,NM2 Resource manager > Now check Nodes decommisioned in /cluster/nodes > Number of decommisioned node is listed as 1 but Table is empty in > /cluster/nodes/decommissioned (detail of Decommision node not shown) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3118) clustering of ATS reader instances
[ https://issues.apache.org/jira/browse/YARN-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298212#comment-14298212 ] Sunil G commented on YARN-3118: --- Thank You Sangjin, I will also have a look on the comment given by Zhijie Zhen on YARN-3047 and will work based on same. > clustering of ATS reader instances > -- > > Key: YARN-3118 > URL: https://issues.apache.org/jira/browse/YARN-3118 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sunil G > > YARN-3047 introduces the ATS reader basically as a single daemon. As a > follow-up, we should consider clustering of ATS reader instances to be able > to handle more traffic volume (large clusters, many use cases, etc.). > It doesn't have to be in phase 1 (maybe for phase 2?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-3118) clustering of ATS reader instances
[ https://issues.apache.org/jira/browse/YARN-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G reassigned YARN-3118: - Assignee: Sunil G > clustering of ATS reader instances > -- > > Key: YARN-3118 > URL: https://issues.apache.org/jira/browse/YARN-3118 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sunil G > > YARN-3047 introduces the ATS reader basically as a single daemon. As a > follow-up, we should consider clustering of ATS reader instances to be able > to handle more traffic volume (large clusters, many use cases, etc.). > It doesn't have to be in phase 1 (maybe for phase 2?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298182#comment-14298182 ] Zhijie Shen commented on YARN-3041: --- [~rkanter], thanks for the patch. I've just attached my data model proposal in the umbrella jira. It seems that we're almost on the same page. Several major differences I've noticed: 1. Metric and event are separate from entity as they are compound too. 2. I suggest using more generalized in/outbound relationship instead of parent-child one. For example, multiple vertex in a dag. 3. I'm wondering if cluster and user are really necessary to be standalone entity instead of one field of an entity. Thoughts? > create the ATS entity/event API > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Robert Kanter > Attachments: YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2928) Application Timeline Server (ATS) next gen: phase 1
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-2928: -- Attachment: Data model proposal v1.pdf Based on the design and previous timeline data model, I drafted and attached a new data model proposal. Please take a look. It doesn't focus on elaborating each individual fields, but showing the major concepts. > Application Timeline Server (ATS) next gen: phase 1 > --- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Vinod Kumar Vavilapalli >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3111) Fix ratio problem on FairScheduler page
[ https://issues.apache.org/jira/browse/YARN-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298157#comment-14298157 ] Peng Zhang commented on YARN-3111: -- I think test failure of "org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore.testFSRMStateStoreClientRetry" is not related with this patch > Fix ratio problem on FairScheduler page > --- > > Key: YARN-3111 > URL: https://issues.apache.org/jira/browse/YARN-3111 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.6.0 >Reporter: Peng Zhang >Assignee: Peng Zhang >Priority: Minor > Fix For: 2.7.0 > > Attachments: YARN-3111.1.patch, YARN-3111.png > > > Found 3 problems on FairScheduler page: > 1. Only compute memory for ratio even when queue schedulingPolicy is DRF. > 2. When min resources is configured larger than real resources, the steady > fair share ratio is so long that it is out the page. > 3. When cluster resources is 0(no nodemanager start), ratio is displayed as > "NaN% used" > Attached image shows the snapshot of above problems. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3075) NodeLabelsManager implementation to retrieve label to node mapping
[ https://issues.apache.org/jira/browse/YARN-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298141#comment-14298141 ] Varun Saxena commented on YARN-3075: Ok... > NodeLabelsManager implementation to retrieve label to node mapping > -- > > Key: YARN-3075 > URL: https://issues.apache.org/jira/browse/YARN-3075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-3075.001.patch, YARN-3075.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3075) NodeLabelsManager implementation to retrieve label to node mapping
[ https://issues.apache.org/jira/browse/YARN-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298142#comment-14298142 ] Varun Saxena commented on YARN-3075: Ok... > NodeLabelsManager implementation to retrieve label to node mapping > -- > > Key: YARN-3075 > URL: https://issues.apache.org/jira/browse/YARN-3075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-3075.001.patch, YARN-3075.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"
[
https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298135#comment-14298135
]
Chris Douglas commented on YARN-1039:
-
bq. That's not necessarily so, there are some cases where the type of life
cycle for an application is important, for example, when determining whether or
not it is open-ended ("service") or a batch process which entails a notion of
progress ("session"), at least for purposes of display.
That's a fair distinction. Would you agree the YARN _scheduler_ should not use
detailed information about progress, task dependencies, or service lifecycles?
If an AM registers with a tag that affects the attributes displayed in
dashboards, then issues like YARN-1079 can be resolved cleanly, as you and
Zhijie propose.
Steve has a point about mixed-mode AMs that run both long and short-lived
containers (e.g., a long-lived service supporting a workflow composed of short
tasks). If it's solely for display, then an enum seems adequate, but I'd like
to better understand the use cases.
> Add parameter for YARN resource requests to indicate "long lived"
> -
>
> Key: YARN-1039
> URL: https://issues.apache.org/jira/browse/YARN-1039
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 3.0.0, 2.1.1-beta
>Reporter: Steve Loughran
>Assignee: Craig Welch
> Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
>
>
> A container request could support a new parameter "long-lived". This could be
> used by a scheduler that would know not to host the service on a transient
> (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived
> containers on the same node
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3118) clustering of ATS reader instances
[ https://issues.apache.org/jira/browse/YARN-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298122#comment-14298122 ] Sunil G commented on YARN-3118: --- Hi Sangjin Does this mean that a third party monitor also needed for managing all reader instances. I could see this as not taken up, and I am interested in taking this. If its not planned, kindly let me know. > clustering of ATS reader instances > -- > > Key: YARN-3118 > URL: https://issues.apache.org/jira/browse/YARN-3118 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee > > YARN-3047 introduces the ATS reader basically as a single daemon. As a > follow-up, we should consider clustering of ATS reader instances to be able > to handle more traffic volume (large clusters, many use cases, etc.). > It doesn't have to be in phase 1 (maybe for phase 2?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3011) NM dies because of the failure of resource localization
[
https://issues.apache.org/jira/browse/YARN-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298118#comment-14298118
]
Varun Saxena commented on YARN-3011:
Thanks [~jianhe] for the review and commit.
> NM dies because of the failure of resource localization
> ---
>
> Key: YARN-3011
> URL: https://issues.apache.org/jira/browse/YARN-3011
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Affects Versions: 2.5.1
>Reporter: Wang Hao
>Assignee: Varun Saxena
> Fix For: 2.7.0
>
> Attachments: YARN-3011.001.patch, YARN-3011.002.patch,
> YARN-3011.003.patch, YARN-3011.004.patch
>
>
> NM dies because of IllegalArgumentException when localize resource.
> 2014-12-29 13:43:58,699 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Downloading public rsrc:{
> hdfs://hadoop002.dx.momo.com:8020/user/hadoop/share/lib/oozie/json-simple-1.1.jar,
> 1416997035456, FILE, null }
> 2014-12-29 13:43:58,699 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Downloading public rsrc:{
> hdfs://hadoop002.dx.momo.com:8020/user/hive/src/final_test_ooize/test_ooize_job1.sql/,
> 1419831474153, FILE, null }
> 2014-12-29 13:43:58,701 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher:
> Error in dispatcher thread
> java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
> at org.apache.hadoop.fs.Path.(Path.java:135)
> at org.apache.hadoop.fs.Path.(Path.java:94)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl.getPathForLocalization(LocalResourcesTrackerImpl.java:420)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$PublicLocalizer.addResource(ResourceLocalizationService.java:758)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:672)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:614)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
>
> at java.lang.Thread.run(Thread.java:745)
> 2014-12-29 13:43:58,701 INFO
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Initializing user hadoop
> 2014-12-29 13:43:58,702 INFO org.apache.hadoop.yarn.event.AsyncDispatcher:
> Exiting, bbye..
> 2014-12-29 13:43:58,704 INFO org.apache.hadoop.mapred.ShuffleHandler: Setting
> connection close header...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3029) FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere
[
https://issues.apache.org/jira/browse/YARN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298116#comment-14298116
]
Varun Saxena commented on YARN-3029:
Thanks [~ste...@apache.org] for review and reporting.
Thanks [~ozawa] for the review and commit.
> FSDownload.unpack() uses local locale for FS case conversion, may not work
> everywhere
> -
>
> Key: YARN-3029
> URL: https://issues.apache.org/jira/browse/YARN-3029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
> Attachments: YARN-3029-003.patch, YARN-3029.001.patch,
> YARN-3029.002.patch
>
>
> {{FSDownload.unpack()}} lower-cases filenames in the local locale before
> looking at extensions for, "tar, "zip", ..
> {code}
> String lowerDst = dst.getName().toLowerCase();
> {code}
> it MUST use LOCALE_EN for the locale, else a file .ZIP won't be recognised as
> a zipfile in a turkish locale cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3089) LinuxContainerExecutor does not handle file arguments to deleteAsUser
[
https://issues.apache.org/jira/browse/YARN-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298111#comment-14298111
]
Hadoop QA commented on YARN-3089:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695421/YARN-3089.v1.txt
against trunk revision f2c9109.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6464//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6464//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6464//console
This message is automatically generated.
> LinuxContainerExecutor does not handle file arguments to deleteAsUser
> -
>
> Key: YARN-3089
> URL: https://issues.apache.org/jira/browse/YARN-3089
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Eric Payne
>Priority: Blocker
> Attachments: YARN-3089.v1.txt
>
>
> YARN-2468 added the deletion of individual logs that are aggregated, but this
> fails to delete log files when the LCE is being used. The LCE native
> executable assumes the paths being passed are paths and the delete fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3100) Make YARN authorization pluggable
[ https://issues.apache.org/jira/browse/YARN-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298090#comment-14298090 ] Jian He commented on YARN-3100: --- bq. I'm basically trying to reconcile the functionality being offered in this JIRA vs. the functionality that we advertise in the service management bits (e.g., http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html#Access_Control_Lists ). Across hdfs and yarn stack, there are basically there types of acl: hdfs-specific ACL, yarn-specific ACL and the common service-level ACL used by both hdfs and YARN which is the link you provided here. What concerns you is the common service level ACL, given it's already being commonly used by YARN and HDFS already, we can definitely do it in a common (but it is out of the scope of this jira as I mentioned before). HDFS-6826 solves hdfs-specific ACL, this jira is to address YARN-specific ACL, and there should be a 3rd jira in common to address the common service-level ACL. Ideally, all ACLs should fit into a single interface. but for yarn and hdfs specific ACL, because YARN and HDFS internal ACL implementation have been differing so much that unifying them is not just a matter of re-factoring but re-designing. That's why I wanted to do it on YARN first to address YARN-specific ACL(which is also what HDFS has been doing to address hdfs-specific ACL) and later on we can have a jira in common to address the common service-level ACL, and in the meantime merging the common part of hdfs-specific acl interface and YARN-specific acl interface into a single common interface. Still, HDFS and YARN will likely have their own specific acl interface left. bq. Adding in the ability to limit by host by merging this functionality would be a large win and actually add functionality that is currently missing to YARN One purpose of this jira is to enable 3rd party tool such as Ranger,Sentry to do authorization for YARN. That is this tool can provide user-defined authorization policy, such as host/ip based authorization policy, time based authorization policy (allowing a user to be able to access between 1:00pm and 2:00pm). And YARN can authorize user based on this policy. I hope this addresses your concern. > Make YARN authorization pluggable > - > > Key: YARN-3100 > URL: https://issues.apache.org/jira/browse/YARN-3100 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-3100.1.patch, YARN-3100.2.patch > > > The goal is to have YARN acl model pluggable so as to integrate other > authorization tool such as Apache Ranger, Sentry. > Currently, we have > - admin ACL > - queue ACL > - application ACL > - time line domain ACL > - service ACL > The proposal is to create a YarnAuthorizationProvider interface. Current > implementation will be the default implementation. Ranger or Sentry plug-in > can implement this interface. > Benefit: > - Unify the code base. With the default implementation, we can get rid of > each specific ACL manager such as AdminAclManager, ApplicationACLsManager, > QueueAclsManager etc. > - Enable Ranger, Sentry to do authorization for YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3075) NodeLabelsManager implementation to retrieve label to node mapping
[ https://issues.apache.org/jira/browse/YARN-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298069#comment-14298069 ] Wangda Tan commented on YARN-3075: -- I meant we can add one, it should have :) > NodeLabelsManager implementation to retrieve label to node mapping > -- > > Key: YARN-3075 > URL: https://issues.apache.org/jira/browse/YARN-3075 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Varun Saxena >Assignee: Varun Saxena > Attachments: YARN-3075.001.patch, YARN-3075.002.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3075) NodeLabelsManager implementation to retrieve label to node mapping
[
https://issues.apache.org/jira/browse/YARN-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298061#comment-14298061
]
Varun Saxena commented on YARN-3075:
[~leftnoteasy],
bq. If we have node id in each Node, you can just loop nms.values() instead of
looping the nms.entrySet(). Just a very minor suggestion.
We do not have node id in Node. Below 3 fields exist in Node class
{code}
public Set labels;
public Resource resource;
public boolean running;
{code}
> NodeLabelsManager implementation to retrieve label to node mapping
> --
>
> Key: YARN-3075
> URL: https://issues.apache.org/jira/browse/YARN-3075
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-3075.001.patch, YARN-3075.002.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3089) LinuxContainerExecutor does not handle file arguments to deleteAsUser
[ https://issues.apache.org/jira/browse/YARN-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-3089: - Attachment: YARN-3089.v1.txt > LinuxContainerExecutor does not handle file arguments to deleteAsUser > - > > Key: YARN-3089 > URL: https://issues.apache.org/jira/browse/YARN-3089 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Eric Payne >Priority: Blocker > Attachments: YARN-3089.v1.txt > > > YARN-2468 added the deletion of individual logs that are aggregated, but this > fails to delete log files when the LCE is being used. The LCE native > executable assumes the paths being passed are paths and the delete fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3104) RM continues to send new AMRM tokens every heartbeat between rolling and activation
[
https://issues.apache.org/jira/browse/YARN-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298028#comment-14298028
]
Jian He commented on YARN-3104:
---
Trying to understand this, do you mean once the connection is authenticated,
take MapReduce as an example, even if MapReduce AM gets the new token and add
the new token into its ugi. But because the connection will not
re-authenticate, it always uses the old token for this comparison
{{nextMasterKey.getMasterKey().getKeyId() != amrmTokenIdentifier.getKeyId()}}
which leads to RM keep sending new tokens ?
> RM continues to send new AMRM tokens every heartbeat between rolling and
> activation
> ---
>
> Key: YARN-3104
> URL: https://issues.apache.org/jira/browse/YARN-3104
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-3104.001.patch
>
>
> When the RM rolls a new AMRM secret, it conveys this to the AMs when it
> notices they are still connected with the old key. However neither the RM
> nor the AM explicitly close the connection or otherwise try to reconnect with
> the new secret. Therefore the RM keeps thinking the AM doesn't have the new
> token on every heartbeat and keeps sending new tokens for the period between
> the key roll and the key activation. Once activated the RM no longer squawks
> in its logs about needing to generate a new token every heartbeat (i.e.:
> second) for every app, but the apps can still be using the old token. The
> token is only checked upon connection to the RM. The apps don't reconnect
> when sent a new token, and the RM doesn't force them to reconnect by closing
> the connection.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3100) Make YARN authorization pluggable
[ https://issues.apache.org/jira/browse/YARN-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298024#comment-14298024 ] Allen Wittenauer commented on YARN-3100: I'm basically trying to reconcile the functionality being offered in this JIRA vs. the functionality that we advertise in the service management bits (e.g., http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ServiceLevelAuth.html#Access_Control_Lists ). To me, the "is a user allowed to do this" needed for YARN is effectively the same functionality. Adding in the ability to limit by host by merging this functionality would be a large win and actually *add* functionality that is currently missing to YARN (esp on the queue side). > Make YARN authorization pluggable > - > > Key: YARN-3100 > URL: https://issues.apache.org/jira/browse/YARN-3100 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-3100.1.patch, YARN-3100.2.patch > > > The goal is to have YARN acl model pluggable so as to integrate other > authorization tool such as Apache Ranger, Sentry. > Currently, we have > - admin ACL > - queue ACL > - application ACL > - time line domain ACL > - service ACL > The proposal is to create a YarnAuthorizationProvider interface. Current > implementation will be the default implementation. Ranger or Sentry plug-in > can implement this interface. > Benefit: > - Unify the code base. With the default implementation, we can get rid of > each specific ACL manager such as AdminAclManager, ApplicationACLsManager, > QueueAclsManager etc. > - Enable Ranger, Sentry to do authorization for YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3041) create the ATS entity/event API
[
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated YARN-3041:
Attachment: YARN-3041.preliminary.001.patch
I think this probably requires more discussion, but here's a preliminary
version of the entities (YARN-3041.preliminary.001.patch). I went through the
design doc and tried to create objects to represent Cluster, Metric, User,
Flow, Flow Run, and Application based on what we had there. Most of the code
is in the parent class, {{TimelineServiceEntity}}.
I know we discussed having these new objects be subtypes of the old
{{TimelineEntity}}, but I wasn't sure the best way to do that; so for now, I
just created a new {{TimelineServiceEntity}} object as the parent. Any
suggestions on this?
I also wasn't sure what Object type the actual metric would be, so I just used
an Object for now. I'm guessing we may have to create different Metric objects
for different metric types.
ping [~zjshen]: you mentioned you were interested in this
> create the ATS entity/event API
> ---
>
> Key: YARN-3041
> URL: https://issues.apache.org/jira/browse/YARN-3041
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Robert Kanter
> Attachments: YARN-3041.preliminary.001.patch
>
>
> Per design in YARN-2928, create the ATS entity and events API.
> Also, as part of this JIRA, create YARN system entities (e.g. cluster, user,
> flow, flow run, YARN app, ...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3047) set up ATS reader with basic request serving structure and lifecycle
[ https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297906#comment-14297906 ] Zhijie Shen commented on YARN-3047: --- Just think it out loudly. Should the reader just be a service instead of being coupled with the web container? Therefore, we can put it into the RM as a light weight web front, start it with a standalone web farm for high available and scalable read service, and even embed it into third-party monitoring tool for integration. > set up ATS reader with basic request serving structure and lifecycle > > > Key: YARN-3047 > URL: https://issues.apache.org/jira/browse/YARN-3047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-3047.001.patch > > > Per design in YARN-2938, set up the ATS reader as a service and implement the > basic structure as a service. It includes lifecycle management, request > serving, and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3100) Make YARN authorization pluggable
[ https://issues.apache.org/jira/browse/YARN-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297903#comment-14297903 ] Jian He commented on YARN-3100: --- bq. It doesn't make sense to all of this base work in YARN when we already know that, even beyond HDFS file-level ACLs, there are all sorts of places that need the same support in common. HDFS-6826 is what HDFS do on their end to make authorization pluggable. I tried to re-use what they have, but the patch has so much hdfs specific bits that won't for YARN. Even with a common interface, hdfs and yarn still require their own interface to take care of their own logic. Stepping back, how much base work it can be ? to yarn, it's very simple, it's just a single interface file which may be used by hdfs. the majority majority work resides in YARN itself. I would be more than happy to put it in common if hdfs would like to use it. Do you suggest a top-down manner, having a single interface sitting in common which works perfectly for hdfs and yarn first in one-go, and then let hdfs and yarn implement their own and no one could ever change the interface ? To me, this is not practical. > Make YARN authorization pluggable > - > > Key: YARN-3100 > URL: https://issues.apache.org/jira/browse/YARN-3100 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-3100.1.patch, YARN-3100.2.patch > > > The goal is to have YARN acl model pluggable so as to integrate other > authorization tool such as Apache Ranger, Sentry. > Currently, we have > - admin ACL > - queue ACL > - application ACL > - time line domain ACL > - service ACL > The proposal is to create a YarnAuthorizationProvider interface. Current > implementation will be the default implementation. Ranger or Sentry plug-in > can implement this interface. > Benefit: > - Unify the code base. With the default implementation, we can get rid of > each specific ACL manager such as AdminAclManager, ApplicationACLsManager, > QueueAclsManager etc. > - Enable Ranger, Sentry to do authorization for YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3047) set up ATS reader with basic request serving structure and lifecycle
[ https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297875#comment-14297875 ] Sangjin Lee commented on YARN-3047: --- Deferring the work of clustering multiple instances to YARN-3118. > set up ATS reader with basic request serving structure and lifecycle > > > Key: YARN-3047 > URL: https://issues.apache.org/jira/browse/YARN-3047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > Attachments: YARN-3047.001.patch > > > Per design in YARN-2938, set up the ATS reader as a service and implement the > basic structure as a service. It includes lifecycle management, request > serving, and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3118) clustering of ATS reader instances
Sangjin Lee created YARN-3118: - Summary: clustering of ATS reader instances Key: YARN-3118 URL: https://issues.apache.org/jira/browse/YARN-3118 Project: Hadoop YARN Issue Type: Sub-task Reporter: Sangjin Lee YARN-3047 introduces the ATS reader basically as a single daemon. As a follow-up, we should consider clustering of ATS reader instances to be able to handle more traffic volume (large clusters, many use cases, etc.). It doesn't have to be in phase 1 (maybe for phase 2?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3043) create ATS configuration, metadata, etc. as part of entities
[ https://issues.apache.org/jira/browse/YARN-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297866#comment-14297866 ] Sangjin Lee commented on YARN-3043: --- I left it a little vague in the doc. I do see metadata largely replacing the primary filters (you could even think of it as renaming it). The semantic is clearer with the name "metadata". I think we want to move away from the primary filters, but I think it's up for debate whether we want to remove it altogether. It partly depends on the migration strategy. > create ATS configuration, metadata, etc. as part of entities > > > Key: YARN-3043 > URL: https://issues.apache.org/jira/browse/YARN-3043 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > > Per design in YARN-2928, create APIs for configuration, metadata, etc. and > integrate them into entities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-545) NodeResourceMonitor and its Impl are emty and may be removed
[
https://issues.apache.org/jira/browse/YARN-545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297868#comment-14297868
]
Hadoop QA commented on YARN-545:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695382/YARN-545.001.patch
against trunk revision 3f982c5.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6463//console
This message is automatically generated.
> NodeResourceMonitor and its Impl are emty and may be removed
>
>
> Key: YARN-545
> URL: https://issues.apache.org/jira/browse/YARN-545
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bikas Saha
>Assignee: Vijay Bhat
>Priority: Minor
> Attachments: YARN-545.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3108) ApplicationHistoryServer doesn't process -D arguments
[ https://issues.apache.org/jira/browse/YARN-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297862#comment-14297862 ] Hudson commented on YARN-3108: -- SUCCESS: Integrated in Hadoop-trunk-Commit #6966 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6966/]) YARN-3108. ApplicationHistoryServer doesn't process -D arguments (Chang Li via jeagles) (jeagles: rev 30a8778c632c0f57cdd005080a470065a60756a8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryServer.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/applicationhistoryservice/TestApplicationHistoryServer.java > ApplicationHistoryServer doesn't process -D arguments > - > > Key: YARN-3108 > URL: https://issues.apache.org/jira/browse/YARN-3108 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chang li >Assignee: chang li > Fix For: 2.7.0 > > Attachments: yarn3108.patch, yarn3108.patch, yarn3108.patch > > > ApplicationHistoryServer doesn't process -D arguments when created, it's nice > to have it to do that -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-545) NodeResourceMonitor and its Impl are emty and may be removed
[ https://issues.apache.org/jira/browse/YARN-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat updated YARN-545: Attachment: YARN-545.001.patch > NodeResourceMonitor and its Impl are emty and may be removed > > > Key: YARN-545 > URL: https://issues.apache.org/jira/browse/YARN-545 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Vijay Bhat >Priority: Minor > Attachments: YARN-545.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-545) NodeResourceMonitor and its Impl are emty and may be removed
[ https://issues.apache.org/jira/browse/YARN-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Bhat reassigned YARN-545: --- Assignee: Vijay Bhat > NodeResourceMonitor and its Impl are emty and may be removed > > > Key: YARN-545 > URL: https://issues.apache.org/jira/browse/YARN-545 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Vijay Bhat >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3100) Make YARN authorization pluggable
[ https://issues.apache.org/jira/browse/YARN-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297831#comment-14297831 ] Allen Wittenauer commented on YARN-3100: In case it isn't obvious, I'm -1 on doing this in YARN first. It doesn't make sense to all of this base work in YARN when we already know that, even beyond HDFS file-level ACLs, there are all sorts of places that need the same support in common. > Make YARN authorization pluggable > - > > Key: YARN-3100 > URL: https://issues.apache.org/jira/browse/YARN-3100 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-3100.1.patch, YARN-3100.2.patch > > > The goal is to have YARN acl model pluggable so as to integrate other > authorization tool such as Apache Ranger, Sentry. > Currently, we have > - admin ACL > - queue ACL > - application ACL > - time line domain ACL > - service ACL > The proposal is to create a YarnAuthorizationProvider interface. Current > implementation will be the default implementation. Ranger or Sentry plug-in > can implement this interface. > Benefit: > - Unify the code base. With the default implementation, we can get rid of > each specific ACL manager such as AdminAclManager, ApplicationACLsManager, > QueueAclsManager etc. > - Enable Ranger, Sentry to do authorization for YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3100) Make YARN authorization pluggable
[
https://issues.apache.org/jira/browse/YARN-3100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297828#comment-14297828
]
Zhijie Shen commented on YARN-3100:
---
bq. I would like to have YARN ready first and then considering merging the
common part into a common interface.
+1 for staring from YARN first.
Here're some comments about the patch.
1. Should {{AccessType}} be the first class citizen too like
{{PrivilegedEntity}}? And there are 1-to-many relationship between
{{PrivilegedEntity}} and {{AccessType}}? For example, Queue -> \{ADD, DELETE,
MODIFY, VIEW\}, App ->
\{SUBMIT, VIEW\}.
2. Do we need to include application as another entity type?
3. No need in {{AdminService.java}} any more.
{code}
private AccessControlList adminAcl;
{code}
4. YarnAuthorizationProvider needs to be annotated? The following java doc is
not as complete as the other methods.
{code}
/**
* Check if the user is an admin.
*/
public abstract boolean isAdmin(UserGroupInformation ugi);
{code}
5. According to the name, signature and implemenation, the method seems not to
be just for verify admin access, but generich. However, it's only used in
AdminService now. After the code change, the method reduces the use case to
only check if it is admin. We may want to avoid the change or refactor the
method name.
{code}
public static UserGroupInformation verifyAccess(
AccessControlList acl, String method, final Log LOG)
throws IOException {
{code}
6. For YarnAuthorizationProvider, it's better to call init() in getInstance(),
such that the user doesn't need to struggle to make sure init() is just called
once and at the first reference. Just think out loudly. Is the singleton
friendly to MiniYARNCluster, where different components reside in a single
process?
> Make YARN authorization pluggable
> -
>
> Key: YARN-3100
> URL: https://issues.apache.org/jira/browse/YARN-3100
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-3100.1.patch, YARN-3100.2.patch
>
>
> The goal is to have YARN acl model pluggable so as to integrate other
> authorization tool such as Apache Ranger, Sentry.
> Currently, we have
> - admin ACL
> - queue ACL
> - application ACL
> - time line domain ACL
> - service ACL
> The proposal is to create a YarnAuthorizationProvider interface. Current
> implementation will be the default implementation. Ranger or Sentry plug-in
> can implement this interface.
> Benefit:
> - Unify the code base. With the default implementation, we can get rid of
> each specific ACL manager such as AdminAclManager, ApplicationACLsManager,
> QueueAclsManager etc.
> - Enable Ranger, Sentry to do authorization for YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (YARN-3117) Conflicting information in the YARN-Docs
[
https://issues.apache.org/jira/browse/YARN-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer moved MAPREDUCE-3227 to YARN-3117:
---
Component/s: (was: documentation)
documentation
Affects Version/s: (was: 0.23.0)
Key: YARN-3117 (was: MAPREDUCE-3227)
Project: Hadoop YARN (was: Hadoop Map/Reduce)
> Conflicting information in the YARN-Docs
>
>
> Key: YARN-3117
> URL: https://issues.apache.org/jira/browse/YARN-3117
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Bhallamudi Venkata Siva Kamesh
> Attachments: MAPREDUCE-3227.patch
>
>
> For building the yarn binaries, information in the *README* file in the
> *hadoop-yarn* is
> {noformat}create runnable binaries after install: mvn assembly:assembly
> -Pnative (combined: mvn clean install assembly:assembly -Pnative){noformat}
> But while executing the above cmd, getting the following error
> {noformat}[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-3:assembly
> (default-cli) on project hadoop-yarn: Error reading assemblies: No assembly
> descriptors found.{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-2979) Unsupported operation exception in message building (YarnProtos)
[ https://issues.apache.org/jira/browse/YARN-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved YARN-2979. -- Resolution: Duplicate > Unsupported operation exception in message building (YarnProtos) > > > Key: YARN-2979 > URL: https://issues.apache.org/jira/browse/YARN-2979 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager >Affects Versions: 2.5.1 >Reporter: Jason Tufo >Assignee: Varun Saxena > Fix For: 2.7.0 > > > java.lang.UnsupportedOperationException > at java.util.AbstractList.add(AbstractList.java:148) > at java.util.AbstractList.add(AbstractList.java:108) > at > com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:330) > at > org.apache.hadoop.yarn.proto.YarnProtos$QueueInfoProto$Builder.addAllApplications(YarnProtos.java:30702) > at > org.apache.hadoop.yarn.api.records.impl.pb.QueueInfoPBImpl.addApplicationsToProto(QueueInfoPBImpl.java:227) > at > org.apache.hadoop.yarn.api.records.impl.pb.QueueInfoPBImpl.mergeLocalToBuilder(QueueInfoPBImpl.java:282) > at > org.apache.hadoop.yarn.api.records.impl.pb.QueueInfoPBImpl.mergeLocalToProto(QueueInfoPBImpl.java:289) > at > org.apache.hadoop.yarn.api.records.impl.pb.QueueInfoPBImpl.getProto(QueueInfoPBImpl.java:157) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoResponsePBImpl.convertToProtoFormat(GetQueueInfoResponsePBImpl.java:128) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoResponsePBImpl.mergeLocalToBuilder(GetQueueInfoResponsePBImpl.java:104) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoResponsePBImpl.mergeLocalToProto(GetQueueInfoResponsePBImpl.java:111) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoResponsePBImpl.getProto(GetQueueInfoResponsePBImpl.java:53) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueInfo(ApplicationClientProtocolPBServiceImpl.java:235) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:333) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (YARN-2979) Unsupported operation exception in message building (YarnProtos)
[ https://issues.apache.org/jira/browse/YARN-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reopened YARN-2979: -- bq. I think current issue status need to be changed as duplicate Agreed. Reopening to resolve as a duplicate of YARN-2978. > Unsupported operation exception in message building (YarnProtos) > > > Key: YARN-2979 > URL: https://issues.apache.org/jira/browse/YARN-2979 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager >Affects Versions: 2.5.1 >Reporter: Jason Tufo >Assignee: Varun Saxena > Fix For: 2.7.0 > > > java.lang.UnsupportedOperationException > at java.util.AbstractList.add(AbstractList.java:148) > at java.util.AbstractList.add(AbstractList.java:108) > at > com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:330) > at > org.apache.hadoop.yarn.proto.YarnProtos$QueueInfoProto$Builder.addAllApplications(YarnProtos.java:30702) > at > org.apache.hadoop.yarn.api.records.impl.pb.QueueInfoPBImpl.addApplicationsToProto(QueueInfoPBImpl.java:227) > at > org.apache.hadoop.yarn.api.records.impl.pb.QueueInfoPBImpl.mergeLocalToBuilder(QueueInfoPBImpl.java:282) > at > org.apache.hadoop.yarn.api.records.impl.pb.QueueInfoPBImpl.mergeLocalToProto(QueueInfoPBImpl.java:289) > at > org.apache.hadoop.yarn.api.records.impl.pb.QueueInfoPBImpl.getProto(QueueInfoPBImpl.java:157) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoResponsePBImpl.convertToProtoFormat(GetQueueInfoResponsePBImpl.java:128) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoResponsePBImpl.mergeLocalToBuilder(GetQueueInfoResponsePBImpl.java:104) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoResponsePBImpl.mergeLocalToProto(GetQueueInfoResponsePBImpl.java:111) > at > org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetQueueInfoResponsePBImpl.getProto(GetQueueInfoResponsePBImpl.java:53) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueInfo(ApplicationClientProtocolPBServiceImpl.java:235) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:333) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3114) It would be better to consider integer(long) overflow when compare the time in DelegationTokenRenewer.
[
https://issues.apache.org/jira/browse/YARN-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297813#comment-14297813
]
Hadoop QA commented on YARN-3114:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695356/YARN-3114.000.patch
against trunk revision 7acce7d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6459//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6459//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6459//console
This message is automatically generated.
> It would be better to consider integer(long) overflow when compare the time
> in DelegationTokenRenewer.
> --
>
> Key: YARN-3114
> URL: https://issues.apache.org/jira/browse/YARN-3114
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: YARN-3114.000.patch
>
>
> It would be better to consider integer(long) overflow when compare the time
> in DelegationTokenRenewer.
> When compare time in DelegationTokenRenewer#DelayedTokenRemovalRunnable to
> cancel token , it will have problem when currentTimeMillis is close to
> Long.MAX_VALUE.
> The safer way to compare time will compare the time difference:
> change
> {code}
> if (e.getValue() < System.currentTimeMillis()) {
> {code}
> to
> {code}
> if (e.getValue() - System.currentTimeMillis() < 0) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-20) More information for "yarn.resourcemanager.webapp.address" in yarn-default.xml
[
https://issues.apache.org/jira/browse/YARN-20?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297811#comment-14297811
]
Hadoop QA commented on YARN-20:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568363/YARN-20.patch
against trunk revision 3f982c5.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6462//console
This message is automatically generated.
> More information for "yarn.resourcemanager.webapp.address" in yarn-default.xml
> --
>
> Key: YARN-20
> URL: https://issues.apache.org/jira/browse/YARN-20
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: documentation, resourcemanager
>Affects Versions: 2.0.0-alpha
>Reporter: Nemon Lou
>Priority: Trivial
> Attachments: YARN-20.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The parameter yarn.resourcemanager.webapp.address in yarn-default.xml is
> in "host:port" format,which is noted in the cluster set up guide
> (http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html).
> When i read though the code,i find "host" format is also supported. In
> "host" format,the port will be random.
> So we may add more documentation in yarn-default.xml for easy understood.
> I will submit a patch if it's helpful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1615) Fix typos in FSSchedulerApp.java
[
https://issues.apache.org/jira/browse/YARN-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297810#comment-14297810
]
Hadoop QA commented on YARN-1615:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623903/YARN-1615.patch
against trunk revision 3f982c5.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6461//console
This message is automatically generated.
> Fix typos in FSSchedulerApp.java
>
>
> Key: YARN-1615
> URL: https://issues.apache.org/jira/browse/YARN-1615
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation, scheduler
>Affects Versions: 2.2.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Trivial
> Labels: newbie
> Attachments: YARN-1615.patch
>
>
> In FSSchedulerApp.java there're 4 typos:
> {code}
>* containers over rack-local or off-switch containers. To acheive this
>* we first only allow node-local assigments for a given prioirty level,
>* then relax the locality threshold once we've had a long enough period
>* without succesfully scheduling. We measure both the number of "missed"
> {code}
> They should be fixed as follows:
> {code}
>* containers over rack-local or off-switch containers. To achieve this
>* we first only allow node-local assignments for a given priority level,
>* then relax the locality threshold once we've had a long enough period
>* without successfully scheduling. We measure both the number of "missed"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2206) Update document for applications REST API response examples
[
https://issues.apache.org/jira/browse/YARN-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297808#comment-14297808
]
Hadoop QA commented on YARN-2206:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652326/YARN-2206.patch
against trunk revision 3f982c5.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6460//console
This message is automatically generated.
> Update document for applications REST API response examples
> ---
>
> Key: YARN-2206
> URL: https://issues.apache.org/jira/browse/YARN-2206
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.4.0
>Reporter: Kenji Kikushima
>Assignee: Kenji Kikushima
>Priority: Minor
> Attachments: YARN-2206.patch
>
>
> In ResourceManagerRest.apt.vm, Applications API responses are missing some
> elements.
> - JSON response should have "applicationType" and "applicationTags".
> - XML response should have "applicationTags".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-308) Improve documentation about what "asks" means in AMRMProtocol
[ https://issues.apache.org/jira/browse/YARN-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-308: -- Fix Version/s: (was: 2.7.0) > Improve documentation about what "asks" means in AMRMProtocol > - > > Key: YARN-308 > URL: https://issues.apache.org/jira/browse/YARN-308 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, documentation, resourcemanager >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-308.patch > > > It's unclear to me from reading the javadoc exactly what "asks" means when > the AM sends a heartbeat to the RM. Is the AM supposed to send a list of all > resources that it is waiting for? Or just inform the RM about new ones that > it wants? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"
[
https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297783#comment-14297783
]
Craig Welch commented on YARN-1039:
---
[~chris.douglas]
bq. YARN shouldn't understand the lifecycle for a service or the
progress/dependencies for task containers
That's not necessarily so, there are some cases where the type of life cycle
for an application is important, for example, when determining whether or not
it is open-ended ("service") or a batch process which entails a notion of
progress ("session"), at least for purposes of display.
I think we need to re scope and clarify this jira a bit so that we can make
progress - there are a number of items in the original problem statement and
subsequent comments which have been taken on elsewhere and so really no longer
make sense to pursue here. Here's an attempt at a breakdown:
bq. This could be used by a scheduler that would know not to host the service
on a transient (cloud: spot priced) node
I think this is now clearly covered by [YARN-796], nodes having qualities
(including operational qualities such as these) is one of the core purposes of
this work, it makes no sense to duplicate it here, and so it should be
de-scoped from this jira
bq. Schedulers could also decide whether or not to allocate multiple long-lived
containers on the same node
As [~ste...@apache.org] mentioned in an earlier comment
[https://issues.apache.org/jira/browse/YARN-1039?focusedCommentId=14038041&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14038041]
affinity / anti-affinity is covered in a more general sense in [YARN-1042].
The above component of this jira is really just such a case, and so it should
be covered with that general solution and dropped from scope as well. There
may be some interest in informing that solution based on a generalized
"service" setting, but to really understand that the affinity approach needs to
be worked out - and I think the affinity approach will really need to
inform/integrate with this rather than the other way around, and integration
should be approached as part of that effort
That leaves nothing, so we can close the jira ;-) Not quite, there were
several things added in comments:
Token management - handled in [YARN-941]
Scheduler hints not related to node categories or anti-affinity (opportunistic
scheduling, etc) - this does strike me as something better handled via the
duration route et all [YARN-2877] [YARN-1051] and not something which needs to
be replicated here
I think that really just leaves the progress bar (and potentially other display
related items). This is covered by [YARN-1079] I suggest, then, that we
either rescope this jira to providing the lifecycle information as an
application tag
[https://issues.apache.org/jira/browse/YARN-1039?focusedCommentId=14039679&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14039679]
as suggested by [~zjshen] early on or close it and cover the work as part of
[YARN-1079]. I originally objected to that approach on the basis that tags
appeared to be a display type feature which did not fit this effort, but if re
scoped as I'm proposing, it becomes such a feature, and I think that approach
is now a good fit.
Thoughts?
> Add parameter for YARN resource requests to indicate "long lived"
> -
>
> Key: YARN-1039
> URL: https://issues.apache.org/jira/browse/YARN-1039
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 3.0.0, 2.1.1-beta
>Reporter: Steve Loughran
>Assignee: Craig Welch
> Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
>
>
> A container request could support a new parameter "long-lived". This could be
> used by a scheduler that would know not to host the service on a transient
> (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived
> containers on the same node
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297753#comment-14297753 ] Thomas Graves commented on YARN-1582: - +1 looks good. Thanks Jason. Feel free to commit. > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch, > YARN-1582.003.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3114) It would be better to consider integer(long) overflow when compare the time in DelegationTokenRenewer.
[
https://issues.apache.org/jira/browse/YARN-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297750#comment-14297750
]
zhihai xu commented on YARN-3114:
-
Thanks for the review. Ok, you are right. The code won't be used until the time
become Long.MAX_VALUE, which may not happen unless the system time is corrupted
due to un-expected error from OS, Hardware, Time server...
My change can work better with error system time and the test code is to prove
this error condition and I should remove it.
> It would be better to consider integer(long) overflow when compare the time
> in DelegationTokenRenewer.
> --
>
> Key: YARN-3114
> URL: https://issues.apache.org/jira/browse/YARN-3114
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: YARN-3114.000.patch
>
>
> It would be better to consider integer(long) overflow when compare the time
> in DelegationTokenRenewer.
> When compare time in DelegationTokenRenewer#DelayedTokenRemovalRunnable to
> cancel token , it will have problem when currentTimeMillis is close to
> Long.MAX_VALUE.
> The safer way to compare time will compare the time difference:
> change
> {code}
> if (e.getValue() < System.currentTimeMillis()) {
> {code}
> to
> {code}
> if (e.getValue() - System.currentTimeMillis() < 0) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297734#comment-14297734 ] Jason Lowe commented on YARN-1582: -- Release audit warning is unrelated, see YARN-3113. The TestSubmitApplicationWithRMHA test failure also appears to be unrelated. > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch, > YARN-1582.003.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3116) We need an assured way to determine if a container is an AM container on NM
Zhijie Shen created YARN-3116: - Summary: We need an assured way to determine if a container is an AM container on NM Key: YARN-3116 URL: https://issues.apache.org/jira/browse/YARN-3116 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen In YARN-3030, to start the per-app aggregator only for a started AM container, we need to determine if the container is an AM container or not from the context in NM (we can do it on RM). This information is missing, such that we worked around to considered the container with ID "_01" as the AM container. Unfortunately, this is neither necessary or sufficient condition. We need to have a way to determine if a container is an AM container on NM. We can add flag to the container object or create an API to do the judgement. Perhaps the distributed AM information may also be useful to YARN-2877. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3115) Work-preserving restarting of per-node aggregator
[ https://issues.apache.org/jira/browse/YARN-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-3115: -- Issue Type: Sub-task (was: Bug) Parent: YARN-2928 > Work-preserving restarting of per-node aggregator > - > > Key: YARN-3115 > URL: https://issues.apache.org/jira/browse/YARN-3115 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > YARN-3030 makes the per-node aggregator work as the aux service of a NM. It > contains the states of the per-app aggregators corresponding to the running > AM containers on this NM. While NM is restarted in work-preserving mode, this > information of per-node aggregator needs to be carried on over restarting too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3115) Work-preserving restarting of per-node aggregator
Zhijie Shen created YARN-3115: - Summary: Work-preserving restarting of per-node aggregator Key: YARN-3115 URL: https://issues.apache.org/jira/browse/YARN-3115 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Reporter: Zhijie Shen Assignee: Zhijie Shen YARN-3030 makes the per-node aggregator work as the aux service of a NM. It contains the states of the per-app aggregators corresponding to the running AM containers on this NM. While NM is restarted in work-preserving mode, this information of per-node aggregator needs to be carried on over restarting too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3114) It would be better to consider integer(long) overflow when compare the time in DelegationTokenRenewer.
[
https://issues.apache.org/jira/browse/YARN-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297681#comment-14297681
]
Jason Lowe commented on YARN-3114:
--
If my math is correct, we won't have this problem for about 292,277 millennia.
IMHO this is effectively a non-issue, as we would be so lucky for this code to
be in use when that becomes a problem. ;-)
On a separate note, the unit test doesn't actually test the code that was
changed. It just asserts that the JVM is performing math correctly with Longs.
The change in DelegationTokenRenewer could later regress while the unit test
continues to pass, which doesn't make it a very effective unit test.
> It would be better to consider integer(long) overflow when compare the time
> in DelegationTokenRenewer.
> --
>
> Key: YARN-3114
> URL: https://issues.apache.org/jira/browse/YARN-3114
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: YARN-3114.000.patch
>
>
> It would be better to consider integer(long) overflow when compare the time
> in DelegationTokenRenewer.
> When compare time in DelegationTokenRenewer#DelayedTokenRemovalRunnable to
> cancel token , it will have problem when currentTimeMillis is close to
> Long.MAX_VALUE.
> The safer way to compare time will compare the time difference:
> change
> {code}
> if (e.getValue() < System.currentTimeMillis()) {
> {code}
> to
> {code}
> if (e.getValue() - System.currentTimeMillis() < 0) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3112) AM restart and keep containers from previous attempts, then new container launch failed
[ https://issues.apache.org/jira/browse/YARN-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297671#comment-14297671 ] Jack Chen commented on YARN-3112: - yeah, i'm trying to find a way to change the umbilical connection in the fly. But before i tackle the real problem, this error comes out, which seems the NMToken are missed for the new appattempt. When i disable the function of keepContainers, there is no problem for the am to restart. maybe there still exist some problem for previous solution? Is there anybody tried that before in a real cluster? > AM restart and keep containers from previous attempts, then new container > launch failed > --- > > Key: YARN-3112 > URL: https://issues.apache.org/jira/browse/YARN-3112 > Project: Hadoop YARN > Issue Type: Bug > Components: applications, resourcemanager >Affects Versions: 2.6.0 > Environment: in real linux cluster >Reporter: Jack Chen > > This error is very similar to YARN-1795, YARN-1839, but i have check the > solution of those jira, the patches are already included in my version. I > think this error is caused by the different NMTokens between old and new > appattempts. New AM has inherited the old tokens from previous AM according > to my configuration (keepContainers=true), so the token for new containers > are replaced by the old one in the NMTokenCache. > 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container > launch failed for container_1422546145900_0001_02_02 : > org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent > for ixk02:47625 > 207 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt > ocolProxy.java:256) > 208 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtoc > olProxy.java:246) > 209 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132) > 210 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401) > 211 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) > 212 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367) > 213 › at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > 214 › at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > 215 › at java.lang.Thread.run(Thread.java:722) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3112) AM restart and keep containers from previous attempts, then new container launch failed
[ https://issues.apache.org/jira/browse/YARN-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297651#comment-14297651 ] Jason Lowe commented on YARN-3112: -- The MapReduce AM supports recovery of completed tasks upon restart, but it does not support reacquiring active containers. Doing so would require the tasks to determine the new address of the subsequent AM attempt so the umbilical connection can be re-established. > AM restart and keep containers from previous attempts, then new container > launch failed > --- > > Key: YARN-3112 > URL: https://issues.apache.org/jira/browse/YARN-3112 > Project: Hadoop YARN > Issue Type: Bug > Components: applications, resourcemanager >Affects Versions: 2.6.0 > Environment: in real linux cluster >Reporter: Jack Chen > > This error is very similar to YARN-1795, YARN-1839, but i have check the > solution of those jira, the patches are already included in my version. I > think this error is caused by the different NMTokens between old and new > appattempts. New AM has inherited the old tokens from previous AM according > to my configuration (keepContainers=true), so the token for new containers > are replaced by the old one in the NMTokenCache. > 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container > launch failed for container_1422546145900_0001_02_02 : > org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent > for ixk02:47625 > 207 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt > ocolProxy.java:256) > 208 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtoc > olProxy.java:246) > 209 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132) > 210 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401) > 211 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) > 212 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367) > 213 › at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > 214 › at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > 215 › at java.lang.Thread.run(Thread.java:722) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3114) It would be better to consider integer(long) overflow when compare the time in DelegationTokenRenewer.
[
https://issues.apache.org/jira/browse/YARN-3114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated YARN-3114:
Attachment: YARN-3114.000.patch
> It would be better to consider integer(long) overflow when compare the time
> in DelegationTokenRenewer.
> --
>
> Key: YARN-3114
> URL: https://issues.apache.org/jira/browse/YARN-3114
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: YARN-3114.000.patch
>
>
> It would be better to consider integer(long) overflow when compare the time
> in DelegationTokenRenewer.
> When compare time in DelegationTokenRenewer#DelayedTokenRemovalRunnable to
> cancel token , it will have problem when currentTimeMillis is close to
> Long.MAX_VALUE.
> The safer way to compare time will compare the time difference:
> change
> {code}
> if (e.getValue() < System.currentTimeMillis()) {
> {code}
> to
> {code}
> if (e.getValue() - System.currentTimeMillis() < 0) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3112) AM restart and keep containers from previous attempts, then new container launch failed
[ https://issues.apache.org/jira/browse/YARN-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297642#comment-14297642 ] Jack Chen commented on YARN-3112: - yes, i'm running mapreduce with the am restart. So AM restart is supported only for other applications except MR? Whether it's feasible to support AM restart in MR? > AM restart and keep containers from previous attempts, then new container > launch failed > --- > > Key: YARN-3112 > URL: https://issues.apache.org/jira/browse/YARN-3112 > Project: Hadoop YARN > Issue Type: Bug > Components: applications, resourcemanager >Affects Versions: 2.6.0 > Environment: in real linux cluster >Reporter: Jack Chen > > This error is very similar to YARN-1795, YARN-1839, but i have check the > solution of those jira, the patches are already included in my version. I > think this error is caused by the different NMTokens between old and new > appattempts. New AM has inherited the old tokens from previous AM according > to my configuration (keepContainers=true), so the token for new containers > are replaced by the old one in the NMTokenCache. > 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container > launch failed for container_1422546145900_0001_02_02 : > org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent > for ixk02:47625 > 207 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt > ocolProxy.java:256) > 208 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtoc > olProxy.java:246) > 209 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132) > 210 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401) > 211 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) > 212 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367) > 213 › at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > 214 › at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > 215 › at java.lang.Thread.run(Thread.java:722) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3114) It would be better to consider integer(long) overflow when compare the time in DelegationTokenRenewer.
zhihai xu created YARN-3114:
---
Summary: It would be better to consider integer(long) overflow
when compare the time in DelegationTokenRenewer.
Key: YARN-3114
URL: https://issues.apache.org/jira/browse/YARN-3114
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
It would be better to consider integer(long) overflow when compare the time in
DelegationTokenRenewer.
When compare time in DelegationTokenRenewer#DelayedTokenRemovalRunnable to
cancel token , it will have problem when currentTimeMillis is close to
Long.MAX_VALUE.
The safer way to compare time will compare the time difference:
change
{code}
if (e.getValue() < System.currentTimeMillis()) {
{code}
to
{code}
if (e.getValue() - System.currentTimeMillis() < 0) {
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3104) RM continues to send new AMRM tokens every heartbeat between rolling and activation
[ https://issues.apache.org/jira/browse/YARN-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297633#comment-14297633 ] Jason Lowe commented on YARN-3104: -- Test failure is unrelated, see YARN-1778. > RM continues to send new AMRM tokens every heartbeat between rolling and > activation > --- > > Key: YARN-3104 > URL: https://issues.apache.org/jira/browse/YARN-3104 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-3104.001.patch > > > When the RM rolls a new AMRM secret, it conveys this to the AMs when it > notices they are still connected with the old key. However neither the RM > nor the AM explicitly close the connection or otherwise try to reconnect with > the new secret. Therefore the RM keeps thinking the AM doesn't have the new > token on every heartbeat and keeps sending new tokens for the period between > the key roll and the key activation. Once activated the RM no longer squawks > in its logs about needing to generate a new token every heartbeat (i.e.: > second) for every app, but the apps can still be using the old token. The > token is only checked upon connection to the RM. The apps don't reconnect > when sent a new token, and the RM doesn't force them to reconnect by closing > the connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1778) TestFSRMStateStore fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297629#comment-14297629
]
Jason Lowe commented on YARN-1778:
--
This test is still failing:
{noformat}
2015-01-28 22:54:42,813 INFO [main] zookeeper.ZKTestCase
(ZKTestCase.java:failed(65)) - FAILED testFSRMStateStoreClientRetry
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore.testFSRMStateStoreClientRetry(TestFSRMStateStore.java:289)
{noformat}
https://builds.apache.org/job/PreCommit-YARN-Build/6442//testReport/ is one
example with more details, and I've seen this fail recently on other precommit
builds.
> TestFSRMStateStore fails on trunk
> -
>
> Key: YARN-1778
> URL: https://issues.apache.org/jira/browse/YARN-1778
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Xuan Gong
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3112) AM restart and keep containers from previous attempts, then new container launch failed
[ https://issues.apache.org/jira/browse/YARN-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297625#comment-14297625 ] Jian He commented on YARN-3112: --- are you running MapReduce? MapReduce doesn't support AM restart with running containers. > AM restart and keep containers from previous attempts, then new container > launch failed > --- > > Key: YARN-3112 > URL: https://issues.apache.org/jira/browse/YARN-3112 > Project: Hadoop YARN > Issue Type: Bug > Components: applications, resourcemanager >Affects Versions: 2.6.0 > Environment: in real linux cluster >Reporter: Jack Chen > > This error is very similar to YARN-1795, YARN-1839, but i have check the > solution of those jira, the patches are already included in my version. I > think this error is caused by the different NMTokens between old and new > appattempts. New AM has inherited the old tokens from previous AM according > to my configuration (keepContainers=true), so the token for new containers > are replaced by the old one in the NMTokenCache. > 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container > launch failed for container_1422546145900_0001_02_02 : > org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent > for ixk02:47625 > 207 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt > ocolProxy.java:256) > 208 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtoc > olProxy.java:246) > 209 › at > org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132) > 210 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401) > 211 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) > 212 › at > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367) > 213 › at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > 214 › at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > 215 › at java.lang.Thread.run(Thread.java:722) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[
https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297621#comment-14297621
]
Hadoop QA commented on YARN-1582:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695336/YARN-1582.003.patch
against trunk revision 89b0749.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6458//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6458//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6458//console
This message is automatically generated.
> Capacity Scheduler: add a maximum-allocation-mb setting per queue
> --
>
> Key: YARN-1582
> URL: https://issues.apache.org/jira/browse/YARN-1582
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 3.0.0, 0.23.10, 2.2.0
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch,
> YARN-1582.003.patch
>
>
> We want to allow certain queues to use larger container sizes while limiting
> other queues to smaller container sizes. Setting it per queue will help
> prevent abuse, help limit the impact of reservations, and allow changes in
> the maximum container size to be rolled out more easily.
> One reason this is needed is more application types are becoming available on
> yarn and certain applications require more memory to run efficiently. While
> we want to allow for that we don't want other applications to abuse that and
> start requesting bigger containers then what they really need.
> Note that we could have this based on application type, but that might not be
> totally accurate either since for example you might want to allow certain
> users on MapReduce to use larger containers, while limiting other users of
> MapReduce to smaller containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3113) Release audit warning for "Sorting icons.psd"
[ https://issues.apache.org/jira/browse/YARN-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-3113: - Summary: Release audit warning for "Sorting icons.psd" (was: some lately checked in changes in "dt-1.9.4/images/Sorting icons.psd" is causing release audit warning) There is a release audit warning for the "Sorting icons.psd" file, but this file was not recently changed. Instead it was triggered by the apache-rat plugin upgrade as part of HADOOP-10574. Updating the JIRA summary accordingly. > Release audit warning for "Sorting icons.psd" > - > > Key: YARN-3113 > URL: https://issues.apache.org/jira/browse/YARN-3113 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: chang li > > some lately checked in changes in "dt-1.9.4/images/Sorting icons.psd" is > causing -1 release audit warning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3108) ApplicationHistoryServer doesn't process -D arguments
[ https://issues.apache.org/jira/browse/YARN-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297498#comment-14297498 ] chang li commented on YARN-3108: have created tracking jira for this problem "https://issues.apache.org/jira/browse/YARN-3113"; > ApplicationHistoryServer doesn't process -D arguments > - > > Key: YARN-3108 > URL: https://issues.apache.org/jira/browse/YARN-3108 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chang li >Assignee: chang li > Attachments: yarn3108.patch, yarn3108.patch, yarn3108.patch > > > ApplicationHistoryServer doesn't process -D arguments when created, it's nice > to have it to do that -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3113) some lately checked in changes in "dt-1.9.4/images/Sorting icons.psd" is causing release audit warning
chang li created YARN-3113: -- Summary: some lately checked in changes in "dt-1.9.4/images/Sorting icons.psd" is causing release audit warning Key: YARN-3113 URL: https://issues.apache.org/jira/browse/YARN-3113 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.7.0 Reporter: chang li some lately checked in changes in "dt-1.9.4/images/Sorting icons.psd" is causing -1 release audit warning -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3108) ApplicationHistoryServer doesn't process -D arguments
[ https://issues.apache.org/jira/browse/YARN-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297492#comment-14297492 ] chang li commented on YARN-3108: This -1 release audit is unrelated to my change, it's caused by some other lately checked in changes. Please ignore this -1 release audit warning and review my patch. Thanks! > ApplicationHistoryServer doesn't process -D arguments > - > > Key: YARN-3108 > URL: https://issues.apache.org/jira/browse/YARN-3108 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chang li >Assignee: chang li > Attachments: yarn3108.patch, yarn3108.patch, yarn3108.patch > > > ApplicationHistoryServer doesn't process -D arguments when created, it's nice > to have it to do that -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3029) FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere
[
https://issues.apache.org/jira/browse/YARN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297446#comment-14297446
]
Hudson commented on YARN-3029:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6963 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6963/])
YARN-3029. FSDownload.unpack() uses local locale for FS case conversion, may
not work everywhere. Contributed by Varun Saxena. (ozawa: rev
7acce7d3648d6f1e45ce280e2147e7dedf5693fc)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java
* hadoop-yarn-project/CHANGES.txt
> FSDownload.unpack() uses local locale for FS case conversion, may not work
> everywhere
> -
>
> Key: YARN-3029
> URL: https://issues.apache.org/jira/browse/YARN-3029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
> Attachments: YARN-3029-003.patch, YARN-3029.001.patch,
> YARN-3029.002.patch
>
>
> {{FSDownload.unpack()}} lower-cases filenames in the local locale before
> looking at extensions for, "tar, "zip", ..
> {code}
> String lowerDst = dst.getName().toLowerCase();
> {code}
> it MUST use LOCALE_EN for the locale, else a file .ZIP won't be recognised as
> a zipfile in a turkish locale cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3112) AM restart and keep containers from previous attempts, then new container launch failed
[ https://issues.apache.org/jira/browse/YARN-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jack Chen updated YARN-3112: Description: This error is very similar to YARN-1795, YARN-1839, but i have check the solution of those jira, the patches are already included in my version. I think this error is caused by the different NMTokens between old and new appattempts. New AM has inherited the old tokens from previous AM according to my configuration (keepContainers=true), so the token for new containers are replaced by the old one in the NMTokenCache. 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1422546145900_0001_02_02 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for ixk02:47625 207 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt ocolProxy.java:256) 208 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtoc olProxy.java:246) 209 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132) 210 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401) 211 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) 212 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367) 213 › at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 214 › at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 215 › at java.lang.Thread.run(Thread.java:722) was: This error is very similar to YARN-713, 1839, but i have check the solution of those jira, the patches are already included in my version. I think this error is caused by the different NMTokens between old and new appattempts. New AM has inherited the old tokens from previous AM according to my configuration (keepContainers=true), so the token for new containers are replaced by the old one in the NMTokenCache. 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1422546145900_0001_02_02 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for ixk02:47625 207 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt ocolProxy.java:256) 208 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtoc olProxy.java:246) 209 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132) 210 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401) 211 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) 212 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367) 213 › at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 214 › at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 215 › at java.lang.Thread.run(Thread.java:722) > AM restart and keep containers from previous attempts, then new container > launch failed > --- > > Key: YARN-3112 > URL: https://issues.apache.org/jira/browse/YARN-3112 > Project: Hadoop YARN > Issue Type: Bug > Components: applications, resourcemanager >Affects Versions: 2.6.0 > Environment: in real linux cluster >Reporter: Jack Chen > > This error is very similar to YARN-1795, YARN-1839, but i have check the > solution of those jira, the patches are already included in my version. I > think this error is caused by the different NMTokens between old and new > appattempts. New AM has inherited the old tokens from previous AM according > to my configuration (keepContainers=true), so the token for new containers > are replaced by the old one in the NMTokenCache. > 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] > org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container > launch failed for container_1422546145900_0
[jira] [Commented] (YARN-3029) FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere
[
https://issues.apache.org/jira/browse/YARN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297433#comment-14297433
]
Tsuyoshi OZAWA commented on YARN-3029:
--
Committed this to trunk and branch-2. Thanks Varun for your contribution and
thanks Steve for the review and sharing. I'll use the command next time.
> FSDownload.unpack() uses local locale for FS case conversion, may not work
> everywhere
> -
>
> Key: YARN-3029
> URL: https://issues.apache.org/jira/browse/YARN-3029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
> Attachments: YARN-3029-003.patch, YARN-3029.001.patch,
> YARN-3029.002.patch
>
>
> {{FSDownload.unpack()}} lower-cases filenames in the local locale before
> looking at extensions for, "tar, "zip", ..
> {code}
> String lowerDst = dst.getName().toLowerCase();
> {code}
> it MUST use LOCALE_EN for the locale, else a file .ZIP won't be recognised as
> a zipfile in a turkish locale cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3112) AM restart and keep containers from previous attempts, then new container launch failed
Jack Chen created YARN-3112: --- Summary: AM restart and keep containers from previous attempts, then new container launch failed Key: YARN-3112 URL: https://issues.apache.org/jira/browse/YARN-3112 Project: Hadoop YARN Issue Type: Bug Components: applications, resourcemanager Affects Versions: 2.6.0 Environment: in real linux cluster Reporter: Jack Chen This error is very similar to YARN-713, 1839, but i have check the solution of those jira, the patches are already included in my version. I think this error is caused by the different NMTokens between old and new appattempts. New AM has inherited the old tokens from previous AM according to my configuration (keepContainers=true), so the token for new containers are replaced by the old one in the NMTokenCache. 206 2015-01-29 10:04:49,603 ERROR [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Container launch failed for container_1422546145900_0001_02_02 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for ixk02:47625 207 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProt ocolProxy.java:256) 208 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.(ContainerManagementProtoc olProxy.java:246) 209 › at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:132) 210 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:401) 211 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138) 212 › at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:367) 213 › at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 214 › at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 215 › at java.lang.Thread.run(Thread.java:722) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3029) FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere
[
https://issues.apache.org/jira/browse/YARN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297408#comment-14297408
]
Steve Loughran commented on YARN-3029:
--
+1
don't worry about trailing whitespace; when applying a patch go
{{git apply --whitespace=fix ... }} and have it do the cleaning for you.
Nobody is likely to complain that the patch applied differs from the final
submission in terms of trailing whitespace.
> FSDownload.unpack() uses local locale for FS case conversion, may not work
> everywhere
> -
>
> Key: YARN-3029
> URL: https://issues.apache.org/jira/browse/YARN-3029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
> Attachments: YARN-3029-003.patch, YARN-3029.001.patch,
> YARN-3029.002.patch
>
>
> {{FSDownload.unpack()}} lower-cases filenames in the local locale before
> looking at extensions for, "tar, "zip", ..
> {code}
> String lowerDst = dst.getName().toLowerCase();
> {code}
> it MUST use LOCALE_EN for the locale, else a file .ZIP won't be recognised as
> a zipfile in a turkish locale cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-1582: - Attachment: YARN-1582.003.patch Patch to address the findbugs warning. The test failures seem unrelated, and they pass for me locally with this patch. > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch, > YARN-1582.003.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2003) Support to process Job priority from Submission Context in AppAttemptAddedSchedulerEvent [RM side]
[ https://issues.apache.org/jira/browse/YARN-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2003: -- Attachment: 0001-YARN-2003.patch Uploading an initial prototype patch for RM changes needed for application priority. This patch is based on existing PriorityLabelManager. As per the discussion, there may be some more changes in that JIRA. Hence this patch will be rebased as per same. Kindly check and share your comments. > Support to process Job priority from Submission Context in > AppAttemptAddedSchedulerEvent [RM side] > -- > > Key: YARN-2003 > URL: https://issues.apache.org/jira/browse/YARN-2003 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-YARN-2003.patch > > > AppAttemptAddedSchedulerEvent should be able to receive the Job Priority from > Submission Context and store. > Later this can be used by Scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3108) ApplicationHistoryServer doesn't process -D arguments
[
https://issues.apache.org/jira/browse/YARN-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297322#comment-14297322
]
Hadoop QA commented on YARN-3108:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695320/yarn3108.patch
against trunk revision 342efa1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6457//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6457//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6457//console
This message is automatically generated.
> ApplicationHistoryServer doesn't process -D arguments
> -
>
> Key: YARN-3108
> URL: https://issues.apache.org/jira/browse/YARN-3108
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: chang li
>Assignee: chang li
> Attachments: yarn3108.patch, yarn3108.patch, yarn3108.patch
>
>
> ApplicationHistoryServer doesn't process -D arguments when created, it's nice
> to have it to do that
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3029) FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere
[
https://issues.apache.org/jira/browse/YARN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297304#comment-14297304
]
Hadoop QA commented on YARN-3029:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695315/YARN-3029-003.patch
against trunk revision 342efa1.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6456//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6456//artifact/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6456//console
This message is automatically generated.
> FSDownload.unpack() uses local locale for FS case conversion, may not work
> everywhere
> -
>
> Key: YARN-3029
> URL: https://issues.apache.org/jira/browse/YARN-3029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
> Attachments: YARN-3029-003.patch, YARN-3029.001.patch,
> YARN-3029.002.patch
>
>
> {{FSDownload.unpack()}} lower-cases filenames in the local locale before
> looking at extensions for, "tar, "zip", ..
> {code}
> String lowerDst = dst.getName().toLowerCase();
> {code}
> it MUST use LOCALE_EN for the locale, else a file .ZIP won't be recognised as
> a zipfile in a turkish locale cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3108) ApplicationHistoryServer doesn't process -D arguments
[ https://issues.apache.org/jira/browse/YARN-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated YARN-3108: --- Attachment: yarn3108.patch replace the hardcoded argument > ApplicationHistoryServer doesn't process -D arguments > - > > Key: YARN-3108 > URL: https://issues.apache.org/jira/browse/YARN-3108 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chang li >Assignee: chang li > Attachments: yarn3108.patch, yarn3108.patch, yarn3108.patch > > > ApplicationHistoryServer doesn't process -D arguments when created, it's nice > to have it to do that -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3029) FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere
[
https://issues.apache.org/jira/browse/YARN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated YARN-3029:
-
Hadoop Flags: Reviewed
> FSDownload.unpack() uses local locale for FS case conversion, may not work
> everywhere
> -
>
> Key: YARN-3029
> URL: https://issues.apache.org/jira/browse/YARN-3029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
> Attachments: YARN-3029-003.patch, YARN-3029.001.patch,
> YARN-3029.002.patch
>
>
> {{FSDownload.unpack()}} lower-cases filenames in the local locale before
> looking at extensions for, "tar, "zip", ..
> {code}
> String lowerDst = dst.getName().toLowerCase();
> {code}
> it MUST use LOCALE_EN for the locale, else a file .ZIP won't be recognised as
> a zipfile in a turkish locale cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3029) FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere
[
https://issues.apache.org/jira/browse/YARN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi OZAWA updated YARN-3029:
-
Attachment: YARN-3029-003.patch
Just removing trailing spaces from rev 002. The content has been reviewed
already. Pending for Jenkins.
> FSDownload.unpack() uses local locale for FS case conversion, may not work
> everywhere
> -
>
> Key: YARN-3029
> URL: https://issues.apache.org/jira/browse/YARN-3029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
> Attachments: YARN-3029-003.patch, YARN-3029.001.patch,
> YARN-3029.002.patch
>
>
> {{FSDownload.unpack()}} lower-cases filenames in the local locale before
> looking at extensions for, "tar, "zip", ..
> {code}
> String lowerDst = dst.getName().toLowerCase();
> {code}
> it MUST use LOCALE_EN for the locale, else a file .ZIP won't be recognised as
> a zipfile in a turkish locale cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[
https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297235#comment-14297235
]
Hadoop QA commented on YARN-1582:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695295/YARN-1582.002.patch
against trunk revision fe2188a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6455//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6455//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6455//console
This message is automatically generated.
> Capacity Scheduler: add a maximum-allocation-mb setting per queue
> --
>
> Key: YARN-1582
> URL: https://issues.apache.org/jira/browse/YARN-1582
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 3.0.0, 0.23.10, 2.2.0
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch
>
>
> We want to allow certain queues to use larger container sizes while limiting
> other queues to smaller container sizes. Setting it per queue will help
> prevent abuse, help limit the impact of reservations, and allow changes in
> the maximum container size to be rolled out more easily.
> One reason this is needed is more application types are becoming available on
> yarn and certain applications require more memory to run efficiently. While
> we want to allow for that we don't want other applications to abuse that and
> start requesting bigger containers then what they really need.
> Note that we could have this based on application type, but that might not be
> totally accurate either since for example you might want to allow certain
> users on MapReduce to use larger containers, while limiting other users of
> MapReduce to smaller containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3029) FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere
[
https://issues.apache.org/jira/browse/YARN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297227#comment-14297227
]
Tsuyoshi OZAWA commented on YARN-3029:
--
+1. There are some lines which include trailing spaces, but I'll remove them.
> FSDownload.unpack() uses local locale for FS case conversion, may not work
> everywhere
> -
>
> Key: YARN-3029
> URL: https://issues.apache.org/jira/browse/YARN-3029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.6.0
>Reporter: Steve Loughran
>Assignee: Varun Saxena
> Attachments: YARN-3029.001.patch, YARN-3029.002.patch
>
>
> {{FSDownload.unpack()}} lower-cases filenames in the local locale before
> looking at extensions for, "tar, "zip", ..
> {code}
> String lowerDst = dst.getName().toLowerCase();
> {code}
> it MUST use LOCALE_EN for the locale, else a file .ZIP won't be recognised as
> a zipfile in a turkish locale cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2428) LCE default banned user list should have yarn
[ https://issues.apache.org/jira/browse/YARN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297213#comment-14297213 ] Hudson commented on YARN-2428: -- FAILURE: Integrated in Hadoop-trunk-Commit #6960 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6960/]) YARN-2428. LCE default banned user list should have yarn (Varun Saxena via aw) (aw: rev 9dd0b7a2ab6538d8f72b004eb97c2750ff3d98dd) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c > LCE default banned user list should have yarn > - > > Key: YARN-2428 > URL: https://issues.apache.org/jira/browse/YARN-2428 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Allen Wittenauer >Assignee: Varun Saxena >Priority: Trivial > Labels: newbie > Fix For: 3.0.0 > > Attachments: YARN-2428.001.patch > > > When task-controller was retrofitted to YARN, the default banned user list > didn't add yarn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-3087) the REST server (web server) for per-node aggregator does not work if it runs inside node manager
[
https://issues.apache.org/jira/browse/YARN-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K reassigned YARN-3087:
---
Assignee: Devaraj K
> the REST server (web server) for per-node aggregator does not work if it runs
> inside node manager
> -
>
> Key: YARN-3087
> URL: https://issues.apache.org/jira/browse/YARN-3087
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Devaraj K
>
> This is related to YARN-3030. YARN-3030 sets up a per-node timeline
> aggregator and the associated REST server. It runs fine as a standalone
> process, but does not work if it runs inside the node manager due to possible
> collisions of servlet mapping.
> Exception:
> {noformat}
> org.apache.hadoop.yarn.webapp.WebAppException: /v2/timeline: controller for
> v2 not found
> at org.apache.hadoop.yarn.webapp.Router.resolveDefault(Router.java:232)
> at org.apache.hadoop.yarn.webapp.Router.resolve(Router.java:140)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:134)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2428) LCE default banned user list should have yarn
[ https://issues.apache.org/jira/browse/YARN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-2428: --- Target Version/s: (was: 2.7.0) > LCE default banned user list should have yarn > - > > Key: YARN-2428 > URL: https://issues.apache.org/jira/browse/YARN-2428 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Allen Wittenauer >Assignee: Varun Saxena >Priority: Trivial > Labels: newbie > Fix For: 3.0.0 > > Attachments: YARN-2428.001.patch > > > When task-controller was retrofitted to YARN, the default banned user list > didn't add yarn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2428) LCE default banned user list should have yarn
[ https://issues.apache.org/jira/browse/YARN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-2428: --- Release Note: The user 'yarn' is no longer allowed to run tasks for security reasons. Hadoop Flags: Incompatible change > LCE default banned user list should have yarn > - > > Key: YARN-2428 > URL: https://issues.apache.org/jira/browse/YARN-2428 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Allen Wittenauer >Assignee: Varun Saxena >Priority: Trivial > Labels: newbie > Fix For: 3.0.0 > > Attachments: YARN-2428.001.patch > > > When task-controller was retrofitted to YARN, the default banned user list > didn't add yarn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2428) LCE default banned user list should have yarn
[ https://issues.apache.org/jira/browse/YARN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-2428: --- Fix Version/s: 3.0.0 > LCE default banned user list should have yarn > - > > Key: YARN-2428 > URL: https://issues.apache.org/jira/browse/YARN-2428 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Allen Wittenauer >Assignee: Varun Saxena >Priority: Trivial > Labels: newbie > Fix For: 3.0.0 > > Attachments: YARN-2428.001.patch > > > When task-controller was retrofitted to YARN, the default banned user list > didn't add yarn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2428) LCE default banned user list should have yarn
[ https://issues.apache.org/jira/browse/YARN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297184#comment-14297184 ] Allen Wittenauer commented on YARN-2428: +1, but I think I'm only going to commit this to trunk since it would be an incompatible change. > LCE default banned user list should have yarn > - > > Key: YARN-2428 > URL: https://issues.apache.org/jira/browse/YARN-2428 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Allen Wittenauer >Assignee: Varun Saxena >Priority: Trivial > Labels: newbie > Attachments: YARN-2428.001.patch > > > When task-controller was retrofitted to YARN, the default banned user list > didn't add yarn. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3108) ApplicationHistoryServer doesn't process -D arguments
[ https://issues.apache.org/jira/browse/YARN-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297153#comment-14297153 ] Jonathan Eagles commented on YARN-3108: --- Thanks for the patch, [~lichangleo]. One thing, can you add a second argument and refer to the YarnConfiguration constants instead of hard coding the argument strings? > ApplicationHistoryServer doesn't process -D arguments > - > > Key: YARN-3108 > URL: https://issues.apache.org/jira/browse/YARN-3108 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chang li >Assignee: chang li > Attachments: yarn3108.patch, yarn3108.patch > > > ApplicationHistoryServer doesn't process -D arguments when created, it's nice > to have it to do that -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3108) ApplicationHistoryServer doesn't process -D arguments
[
https://issues.apache.org/jira/browse/YARN-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297117#comment-14297117
]
Hadoop QA commented on YARN-3108:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695293/yarn3108.patch
against trunk revision 03a5e04.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6454//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6454//console
This message is automatically generated.
> ApplicationHistoryServer doesn't process -D arguments
> -
>
> Key: YARN-3108
> URL: https://issues.apache.org/jira/browse/YARN-3108
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: chang li
>Assignee: chang li
> Attachments: yarn3108.patch, yarn3108.patch
>
>
> ApplicationHistoryServer doesn't process -D arguments when created, it's nice
> to have it to do that
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3021) YARN's delegation-token handling disallows certain trust setups to operate properly
[ https://issues.apache.org/jira/browse/YARN-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297112#comment-14297112 ] Yongjun Zhang commented on YARN-3021: - Hi [~vinodkv], Thanks for your earlier comments and suggestions, would you please help taking a look at the patch? Thanks a lot! > YARN's delegation-token handling disallows certain trust setups to operate > properly > --- > > Key: YARN-3021 > URL: https://issues.apache.org/jira/browse/YARN-3021 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 2.3.0 >Reporter: Harsh J > Attachments: YARN-3021.001.patch, YARN-3021.patch > > > Consider this scenario of 3 realms: A, B and COMMON, where A trusts COMMON, > and B trusts COMMON (one way trusts both), and both A and B run HDFS + YARN > clusters. > Now if one logs in with a COMMON credential, and runs a job on A's YARN that > needs to access B's HDFS (such as a DistCp), the operation fails in the RM, > as it attempts a renewDelegationToken(…) synchronously during application > submission (to validate the managed token before it adds it to a scheduler > for automatic renewal). The call obviously fails cause B realm will not trust > A's credentials (here, the RM's principal is the renewer). > In the 1.x JobTracker the same call is present, but it is done asynchronously > and once the renewal attempt failed we simply ceased to schedule any further > attempts of renewals, rather than fail the job immediately. > We should change the logic such that we attempt the renewal but go easy on > the failure and skip the scheduling alone, rather than bubble back an error > to the client, failing the app submission. This way the old behaviour is > retained. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-1582) Capacity Scheduler: add a maximum-allocation-mb setting per queue
[ https://issues.apache.org/jira/browse/YARN-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-1582: - Attachment: YARN-1582.002.patch Updating Tom's patch to trunk and also adding support for vcores. > Capacity Scheduler: add a maximum-allocation-mb setting per queue > -- > > Key: YARN-1582 > URL: https://issues.apache.org/jira/browse/YARN-1582 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: YARN-1582-branch-0.23.patch, YARN-1582.002.patch > > > We want to allow certain queues to use larger container sizes while limiting > other queues to smaller container sizes. Setting it per queue will help > prevent abuse, help limit the impact of reservations, and allow changes in > the maximum container size to be rolled out more easily. > One reason this is needed is more application types are becoming available on > yarn and certain applications require more memory to run efficiently. While > we want to allow for that we don't want other applications to abuse that and > start requesting bigger containers then what they really need. > Note that we could have this based on application type, but that might not be > totally accurate either since for example you might want to allow certain > users on MapReduce to use larger containers, while limiting other users of > MapReduce to smaller containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3108) ApplicationHistoryServer doesn't process -D arguments
[ https://issues.apache.org/jira/browse/YARN-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated YARN-3108: --- Attachment: yarn3108.patch > ApplicationHistoryServer doesn't process -D arguments > - > > Key: YARN-3108 > URL: https://issues.apache.org/jira/browse/YARN-3108 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chang li >Assignee: chang li > Attachments: yarn3108.patch, yarn3108.patch > > > ApplicationHistoryServer doesn't process -D arguments when created, it's nice > to have it to do that -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3103) AMRMClientImpl does not update AMRM token properly
[ https://issues.apache.org/jira/browse/YARN-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297006#comment-14297006 ] Hudson commented on YARN-3103: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2039 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2039/]) YARN-3103. AMRMClientImpl does not update AMRM token properly. Contributed by Jason Lowe (jianhe: rev 6d2bdbd7dab179dfb4f19bb41809e97f1db88c6b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/CHANGES.txt > AMRMClientImpl does not update AMRM token properly > -- > > Key: YARN-3103 > URL: https://issues.apache.org/jira/browse/YARN-3103 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3103.001.patch > > > AMRMClientImpl.updateAMRMToken updates the token service _before_ storing it > to the credentials, so the token is mapped using the newly updated service > rather than the empty service that was used when the RM created the original > AMRM token. This leads to two AMRM tokens in the credentials and can still > fail if the AMRMTokenSelector picks the wrong one. > In addition the AMRMClientImpl grabs the login user rather than the current > user when security is enabled, so it's likely the UGI being updated is not > the UGI that will be used when reconnecting to the RM. > The end result is that AMs can fail with invalid token errors when trying to > reconnect to an RM after a new AMRM secret has been activated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3079) Scheduler should also update maximumAllocation when updateNodeResource.
[ https://issues.apache.org/jira/browse/YARN-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14297000#comment-14297000 ] Hudson commented on YARN-3079: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2039 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2039/]) YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) (wangda: rev 7882bc0f1433ae73361cab4207eb0c15abee4586) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java > Scheduler should also update maximumAllocation when updateNodeResource. > --- > > Key: YARN-3079 > URL: https://issues.apache.org/jira/browse/YARN-3079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.7.0 > > Attachments: YARN-3079.000.patch, YARN-3079.001.patch, > YARN-3079.002.patch, YARN-3079.003.patch, YARN-3079.004.patch > > > Scheduler should also update maximumAllocation when updateNodeResource. > Otherwise even the node resource is changed by > AdminService#updateNodeResource, maximumAllocation won't be changed. > Also RMNodeReconnectEvent called from > ResourceTrackerService#registerNodeManager will also trigger > AbstractYarnScheduler#updateNodeResource being called. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2005) Blacklisting support for scheduling AMs
[ https://issues.apache.org/jira/browse/YARN-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296970#comment-14296970 ] Sunil G commented on YARN-2005: --- bq. we could implement an app-specific blacklisting approach similar to what is done by the MapReduce AM today. +1 Yes. I also feel this can be the first step. Without much complicating, a step can be made forward. For that, we need changes in RMAppImpl transitions to keep track of host which previous failed attempt ran. Then pass that to Scheduler as an exclude list only for AM schedule. Could I work on this, and I can share a prototype soon. Then later it can made to production standard by splitting to appropriate subjira. > Blacklisting support for scheduling AMs > --- > > Key: YARN-2005 > URL: https://issues.apache.org/jira/browse/YARN-2005 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 0.23.10, 2.4.0 >Reporter: Jason Lowe > > It would be nice if the RM supported blacklisting a node for an AM launch > after the same node fails a configurable number of AM attempts. This would > be similar to the blacklisting support for scheduling task attempts in the > MapReduce AM but for scheduling AM attempts on the RM side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2005) Blacklisting support for scheduling AMs
[ https://issues.apache.org/jira/browse/YARN-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296953#comment-14296953 ] Jason Lowe commented on YARN-2005: -- As I mentioned earlier, as a first step I think we could implement an app-specific blacklisting approach similar to what is done by the MapReduce AM today. We would track, per application, the nodes that have failed an AM attempt and refuse to launch subsequent AM attempts for that application on those nodes. If we want to keep it really simple, we could just do literally that. From there we can sprinkle additional logic to make it a bit more sophisticated, e.g.: having the blacklisting auto-disable when the percentage of blacklisted nodes compared to the total active nodes is above some threshold and/or the app has waited some amount of time for an AM container for the next attempt. > Blacklisting support for scheduling AMs > --- > > Key: YARN-2005 > URL: https://issues.apache.org/jira/browse/YARN-2005 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 0.23.10, 2.4.0 >Reporter: Jason Lowe > > It would be nice if the RM supported blacklisting a node for an AM launch > after the same node fails a configurable number of AM attempts. This would > be similar to the blacklisting support for scheduling task attempts in the > MapReduce AM but for scheduling AM attempts on the RM side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2005) Blacklisting support for scheduling AMs
[ https://issues.apache.org/jira/browse/YARN-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296936#comment-14296936 ] Sunil G commented on YARN-2005: --- bq. I've seen use a different app name each time they submit Yes. This is a real valid point in a production cluster. I hope the user will be same for these, hence user can play a bigger role in identifying such problematic apps. as mentioned in point 2, * failed app can be restricted to run in same node on a reattempt scenario * If an app is failed for a given user, and same user submitting another application, its good to scheduler that also in a different node. bq.but the biggest problem we're seeing probably doesn't need anything that fancy to solve 80% of the cases we see. I agree that a simple logic can be shaped up first, and can see the feasibility. Then an analysis on how much it can really help. After that its better to go for complexity. Kindly suggests your thoughts on same. > Blacklisting support for scheduling AMs > --- > > Key: YARN-2005 > URL: https://issues.apache.org/jira/browse/YARN-2005 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 0.23.10, 2.4.0 >Reporter: Jason Lowe > > It would be nice if the RM supported blacklisting a node for an AM launch > after the same node fails a configurable number of AM attempts. This would > be similar to the blacklisting support for scheduling task attempts in the > MapReduce AM but for scheduling AM attempts on the RM side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3079) Scheduler should also update maximumAllocation when updateNodeResource.
[ https://issues.apache.org/jira/browse/YARN-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296927#comment-14296927 ] Hudson commented on YARN-3079: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2020 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2020/]) YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) (wangda: rev 7882bc0f1433ae73361cab4207eb0c15abee4586) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java > Scheduler should also update maximumAllocation when updateNodeResource. > --- > > Key: YARN-3079 > URL: https://issues.apache.org/jira/browse/YARN-3079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.7.0 > > Attachments: YARN-3079.000.patch, YARN-3079.001.patch, > YARN-3079.002.patch, YARN-3079.003.patch, YARN-3079.004.patch > > > Scheduler should also update maximumAllocation when updateNodeResource. > Otherwise even the node resource is changed by > AdminService#updateNodeResource, maximumAllocation won't be changed. > Also RMNodeReconnectEvent called from > ResourceTrackerService#registerNodeManager will also trigger > AbstractYarnScheduler#updateNodeResource being called. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3103) AMRMClientImpl does not update AMRM token properly
[ https://issues.apache.org/jira/browse/YARN-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296934#comment-14296934 ] Hudson commented on YARN-3103: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2020 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2020/]) YARN-3103. AMRMClientImpl does not update AMRM token properly. Contributed by Jason Lowe (jianhe: rev 6d2bdbd7dab179dfb4f19bb41809e97f1db88c6b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/CHANGES.txt > AMRMClientImpl does not update AMRM token properly > -- > > Key: YARN-3103 > URL: https://issues.apache.org/jira/browse/YARN-3103 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3103.001.patch > > > AMRMClientImpl.updateAMRMToken updates the token service _before_ storing it > to the credentials, so the token is mapped using the newly updated service > rather than the empty service that was used when the RM created the original > AMRM token. This leads to two AMRM tokens in the credentials and can still > fail if the AMRMTokenSelector picks the wrong one. > In addition the AMRMClientImpl grabs the login user rather than the current > user when security is enabled, so it's likely the UGI being updated is not > the UGI that will be used when reconnecting to the RM. > The end result is that AMs can fail with invalid token errors when trying to > reconnect to an RM after a new AMRM secret has been activated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3103) AMRMClientImpl does not update AMRM token properly
[ https://issues.apache.org/jira/browse/YARN-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296900#comment-14296900 ] Hudson commented on YARN-3103: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #85 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/85/]) YARN-3103. AMRMClientImpl does not update AMRM token properly. Contributed by Jason Lowe (jianhe: rev 6d2bdbd7dab179dfb4f19bb41809e97f1db88c6b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/CHANGES.txt > AMRMClientImpl does not update AMRM token properly > -- > > Key: YARN-3103 > URL: https://issues.apache.org/jira/browse/YARN-3103 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3103.001.patch > > > AMRMClientImpl.updateAMRMToken updates the token service _before_ storing it > to the credentials, so the token is mapped using the newly updated service > rather than the empty service that was used when the RM created the original > AMRM token. This leads to two AMRM tokens in the credentials and can still > fail if the AMRMTokenSelector picks the wrong one. > In addition the AMRMClientImpl grabs the login user rather than the current > user when security is enabled, so it's likely the UGI being updated is not > the UGI that will be used when reconnecting to the RM. > The end result is that AMs can fail with invalid token errors when trying to > reconnect to an RM after a new AMRM secret has been activated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3079) Scheduler should also update maximumAllocation when updateNodeResource.
[ https://issues.apache.org/jira/browse/YARN-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296893#comment-14296893 ] Hudson commented on YARN-3079: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #85 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/85/]) YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) (wangda: rev 7882bc0f1433ae73361cab4207eb0c15abee4586) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java > Scheduler should also update maximumAllocation when updateNodeResource. > --- > > Key: YARN-3079 > URL: https://issues.apache.org/jira/browse/YARN-3079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.7.0 > > Attachments: YARN-3079.000.patch, YARN-3079.001.patch, > YARN-3079.002.patch, YARN-3079.003.patch, YARN-3079.004.patch > > > Scheduler should also update maximumAllocation when updateNodeResource. > Otherwise even the node resource is changed by > AdminService#updateNodeResource, maximumAllocation won't be changed. > Also RMNodeReconnectEvent called from > ResourceTrackerService#registerNodeManager will also trigger > AbstractYarnScheduler#updateNodeResource being called. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3103) AMRMClientImpl does not update AMRM token properly
[ https://issues.apache.org/jira/browse/YARN-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296809#comment-14296809 ] Hudson commented on YARN-3103: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #89 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/89/]) YARN-3103. AMRMClientImpl does not update AMRM token properly. Contributed by Jason Lowe (jianhe: rev 6d2bdbd7dab179dfb4f19bb41809e97f1db88c6b) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java > AMRMClientImpl does not update AMRM token properly > -- > > Key: YARN-3103 > URL: https://issues.apache.org/jira/browse/YARN-3103 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3103.001.patch > > > AMRMClientImpl.updateAMRMToken updates the token service _before_ storing it > to the credentials, so the token is mapped using the newly updated service > rather than the empty service that was used when the RM created the original > AMRM token. This leads to two AMRM tokens in the credentials and can still > fail if the AMRMTokenSelector picks the wrong one. > In addition the AMRMClientImpl grabs the login user rather than the current > user when security is enabled, so it's likely the UGI being updated is not > the UGI that will be used when reconnecting to the RM. > The end result is that AMs can fail with invalid token errors when trying to > reconnect to an RM after a new AMRM secret has been activated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3079) Scheduler should also update maximumAllocation when updateNodeResource.
[ https://issues.apache.org/jira/browse/YARN-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296803#comment-14296803 ] Hudson commented on YARN-3079: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #89 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/89/]) YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) (wangda: rev 7882bc0f1433ae73361cab4207eb0c15abee4586) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java > Scheduler should also update maximumAllocation when updateNodeResource. > --- > > Key: YARN-3079 > URL: https://issues.apache.org/jira/browse/YARN-3079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.7.0 > > Attachments: YARN-3079.000.patch, YARN-3079.001.patch, > YARN-3079.002.patch, YARN-3079.003.patch, YARN-3079.004.patch > > > Scheduler should also update maximumAllocation when updateNodeResource. > Otherwise even the node resource is changed by > AdminService#updateNodeResource, maximumAllocation won't be changed. > Also RMNodeReconnectEvent called from > ResourceTrackerService#registerNodeManager will also trigger > AbstractYarnScheduler#updateNodeResource being called. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3103) AMRMClientImpl does not update AMRM token properly
[ https://issues.apache.org/jira/browse/YARN-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296735#comment-14296735 ] Hudson commented on YARN-3103: -- FAILURE: Integrated in Hadoop-Yarn-trunk #822 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/822/]) YARN-3103. AMRMClientImpl does not update AMRM token properly. Contributed by Jason Lowe (jianhe: rev 6d2bdbd7dab179dfb4f19bb41809e97f1db88c6b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/CHANGES.txt > AMRMClientImpl does not update AMRM token properly > -- > > Key: YARN-3103 > URL: https://issues.apache.org/jira/browse/YARN-3103 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3103.001.patch > > > AMRMClientImpl.updateAMRMToken updates the token service _before_ storing it > to the credentials, so the token is mapped using the newly updated service > rather than the empty service that was used when the RM created the original > AMRM token. This leads to two AMRM tokens in the credentials and can still > fail if the AMRMTokenSelector picks the wrong one. > In addition the AMRMClientImpl grabs the login user rather than the current > user when security is enabled, so it's likely the UGI being updated is not > the UGI that will be used when reconnecting to the RM. > The end result is that AMs can fail with invalid token errors when trying to > reconnect to an RM after a new AMRM secret has been activated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3079) Scheduler should also update maximumAllocation when updateNodeResource.
[ https://issues.apache.org/jira/browse/YARN-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296728#comment-14296728 ] Hudson commented on YARN-3079: -- FAILURE: Integrated in Hadoop-Yarn-trunk #822 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/822/]) YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) (wangda: rev 7882bc0f1433ae73361cab4207eb0c15abee4586) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java * hadoop-yarn-project/CHANGES.txt > Scheduler should also update maximumAllocation when updateNodeResource. > --- > > Key: YARN-3079 > URL: https://issues.apache.org/jira/browse/YARN-3079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.7.0 > > Attachments: YARN-3079.000.patch, YARN-3079.001.patch, > YARN-3079.002.patch, YARN-3079.003.patch, YARN-3079.004.patch > > > Scheduler should also update maximumAllocation when updateNodeResource. > Otherwise even the node resource is changed by > AdminService#updateNodeResource, maximumAllocation won't be changed. > Also RMNodeReconnectEvent called from > ResourceTrackerService#registerNodeManager will also trigger > AbstractYarnScheduler#updateNodeResource being called. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3103) AMRMClientImpl does not update AMRM token properly
[ https://issues.apache.org/jira/browse/YARN-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296724#comment-14296724 ] Hudson commented on YARN-3103: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #88 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/88/]) YARN-3103. AMRMClientImpl does not update AMRM token properly. Contributed by Jason Lowe (jianhe: rev 6d2bdbd7dab179dfb4f19bb41809e97f1db88c6b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/CHANGES.txt > AMRMClientImpl does not update AMRM token properly > -- > > Key: YARN-3103 > URL: https://issues.apache.org/jira/browse/YARN-3103 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Fix For: 2.7.0 > > Attachments: YARN-3103.001.patch > > > AMRMClientImpl.updateAMRMToken updates the token service _before_ storing it > to the credentials, so the token is mapped using the newly updated service > rather than the empty service that was used when the RM created the original > AMRM token. This leads to two AMRM tokens in the credentials and can still > fail if the AMRMTokenSelector picks the wrong one. > In addition the AMRMClientImpl grabs the login user rather than the current > user when security is enabled, so it's likely the UGI being updated is not > the UGI that will be used when reconnecting to the RM. > The end result is that AMs can fail with invalid token errors when trying to > reconnect to an RM after a new AMRM secret has been activated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3079) Scheduler should also update maximumAllocation when updateNodeResource.
[ https://issues.apache.org/jira/browse/YARN-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296717#comment-14296717 ] Hudson commented on YARN-3079: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #88 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/88/]) YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) (wangda: rev 7882bc0f1433ae73361cab4207eb0c15abee4586) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/CHANGES.txt > Scheduler should also update maximumAllocation when updateNodeResource. > --- > > Key: YARN-3079 > URL: https://issues.apache.org/jira/browse/YARN-3079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.7.0 > > Attachments: YARN-3079.000.patch, YARN-3079.001.patch, > YARN-3079.002.patch, YARN-3079.003.patch, YARN-3079.004.patch > > > Scheduler should also update maximumAllocation when updateNodeResource. > Otherwise even the node resource is changed by > AdminService#updateNodeResource, maximumAllocation won't be changed. > Also RMNodeReconnectEvent called from > ResourceTrackerService#registerNodeManager will also trigger > AbstractYarnScheduler#updateNodeResource being called. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
