[jira] [Commented] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.
[ https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140910#comment-15140910 ] Lei Guo commented on YARN-1051: --- [~curino], for enterprise customer, the scheduling policy will be complicate. How this planner to satisfy complex scheduling policy other than FIFO? As [~acmurthy] asked earlier, the priority based scheduling is one basic case on scheduling policy, what's the best practice for this? > YARN Admission Control/Planner: enhancing the resource allocation model with > time. > -- > > Key: YARN-1051 > URL: https://issues.apache.org/jira/browse/YARN-1051 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, resourcemanager, scheduler >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.6.0 > > Attachments: YARN-1051-design.pdf, YARN-1051.1.patch, > YARN-1051.patch, curino_MSR-TR-2013-108.pdf, socc14-paper15.pdf, > techreport.pdf > > > In this umbrella JIRA we propose to extend the YARN RM to handle time > explicitly, allowing users to "reserve" capacity over time. This is an > important step towards SLAs, long-running services, workflows, and helps for > gang scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4655) Log uncaught exceptions/errors in various thread pools in the nodemanger
[ https://issues.apache.org/jira/browse/YARN-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140991#comment-15140991 ] Varun Vasudev commented on YARN-4655: - +1. I'll commit this tomorrow if no one objects. > Log uncaught exceptions/errors in various thread pools in the nodemanger > > > Key: YARN-4655 > URL: https://issues.apache.org/jira/browse/YARN-4655 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4655.001.patch, YARN-4655.002.patch > > > For a detailed description, please see HADOOP-12748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4653) Document YARN security model from the perspective of Application Developers
[ https://issues.apache.org/jira/browse/YARN-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated YARN-4653: - Attachment: YARN-4653-004.patch Patch 004; write down my understanding of AM token renewal (based on feedback in this JIRA), some final review of the text > Document YARN security model from the perspective of Application Developers > --- > > Key: YARN-4653 > URL: https://issues.apache.org/jira/browse/YARN-4653 > Project: Hadoop YARN > Issue Type: Task > Components: site >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: YARN-4653-001.patch, YARN-4653-002.patch, > YARN-4653-003.patch, YARN-4653-004.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > What YARN apps need to do for security today is generally copied direct from > distributed shell, with a bit of [ill-informed > superstition|https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/yarn.html] > being the sole prose. > We need a normative document in the YARN site covering > # the needs for YARN security > # token creation for AM launch > # how the RM gets involved > # token propagation on container launch > # token renewal strategies > # How to get tokens for other apps like HBase and Hive. > # how to work under OOzie > Perhaps the WritingYarnApplications.md doc is updated, otherwise why not just > link to the relevant bit of the distributed shell client on github for a > guarantee of staying up to date? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4653) Document YARN security model from the perspective of Application Developers
[
https://issues.apache.org/jira/browse/YARN-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140828#comment-15140828
]
Hadoop QA commented on YARN-4653:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 15 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 3m 11s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787262/YARN-4653-004.patch |
| JIRA Issue | YARN-4653 |
| Optional Tests | asflicense mvnsite xml |
| uname | Linux d24823a12d87 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e9a6226 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/10545/artifact/patchprocess/whitespace-eol.txt
|
| modules | C: hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: . |
| Max memory used | 52MB |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/10545/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Document YARN security model from the perspective of Application Developers
> ---
>
> Key: YARN-4653
> URL: https://issues.apache.org/jira/browse/YARN-4653
> Project: Hadoop YARN
> Issue Type: Task
> Components: site
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: YARN-4653-001.patch, YARN-4653-002.patch,
> YARN-4653-003.patch, YARN-4653-004.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> What YARN apps need to do for security today is generally copied direct from
> distributed shell, with a bit of [ill-informed
> superstition|https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/yarn.html]
> being the sole prose.
> We need a normative document in the YARN site covering
> # the needs for YARN security
> # token creation for AM launch
> # how the RM gets involved
> # token propagation on container launch
> # token renewal strategies
> # How to get tokens for other apps like HBase and Hive.
> # how to work under OOzie
> Perhaps the WritingYarnApplications.md doc is updated, otherwise why not just
> link to the relevant bit of the distributed shell client on github for a
> guarantee of staying up to date?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4138) Roll back container resource allocation after resource increase token expires
[ https://issues.apache.org/jira/browse/YARN-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140973#comment-15140973 ] Jian He commented on YARN-4138: --- patch looks good to me, [~sandflee], do you have any more comments ? > Roll back container resource allocation after resource increase token expires > - > > Key: YARN-4138 > URL: https://issues.apache.org/jira/browse/YARN-4138 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: MENG DING >Assignee: MENG DING > Attachments: YARN-4138-YARN-1197.1.patch, > YARN-4138-YARN-1197.2.patch, YARN-4138.3.patch, YARN-4138.4.patch, > YARN-4138.5.patch, YARN-4138.6.patch, YARN-4138.7.patch > > > In YARN-1651, after container resource increase token expires, the running > container is killed. > This ticket will change the behavior such that when a container resource > increase token expires, the resource allocation of the container will be > reverted back to the value before the increase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4412) Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers
[ https://issues.apache.org/jira/browse/YARN-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141241#comment-15141241 ] Arun Suresh commented on YARN-4412: --- [~curino], is it possible to give this a review ? > Create ClusterMonitor to compute ordered list of preferred NMs for > OPPORTUNITIC containers > -- > > Key: YARN-4412 > URL: https://issues.apache.org/jira/browse/YARN-4412 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-4412-yarn-2877.v1.patch, > YARN-4412-yarn-2877.v2.patch > > > Introduce a Cluster Monitor that aggregates load information from individual > Node Managers and computes an ordered list of preferred Node managers to be > used as target Nodes for OPPORTUNISTIC container allocations. > This list can be pushed out to the Node Manager (specifically the AMRMProxy > running on the Node) via the Allocate Response. This will be used to make > local Scheduling decisions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1051) YARN Admission Control/Planner: enhancing the resource allocation model with time.
[ https://issues.apache.org/jira/browse/YARN-1051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141266#comment-15141266 ] Carlo Curino commented on YARN-1051: [~grey], I suggest you to read the attached techreport for full context, but let me try to summarize the ideas here. *General Idea* The reservation system receives reservation requests from users over a period of time. Note that each reservation can request resources much ahead of time (e.g., I need 10 containers for 1 hour tomorrow sometime between 3pm and 6pm). The planner will try to "fit" all these reservation in the plan agenda, while respecting the user constraints (e.g., amount of resources and start_time/deadline) and the physical constraints of the plan (which is a "queue", and thus has access to a portion of the cluster capacity). The APIs exposed to the users allow them to expose their flexibility (e.g., for a map-only job I can express the fact that I can run with up to 10 parallel containers, but also 1 container at a time), this allows the plan to fit more jobs by "deforming them". A side effect of this is that we can provide support for gang-semantics (e.g., I need 10 concurrent containers for 1 h). The key intuition is that each job might temporarily use a large amount of resources, but we control very explicitly when it should yield resources back to other jobs. This explicit time-multiplexing gives very strong guarantees to each job (i.e., if the reservation was accepted you will get your resources), but allows us to densely pack the cluster agenda (and thus get high utilization / high ROI). Moreover, best-effort jobs can be run on separate queues with the standard set of scheduling invariant provided by FairScheduler/CapacityScheduler. *SharingPolicy* Another interesting area in which enterprise settings can extend/innovate is the choice of "SharingPolicy". The SharingPolicy is a way for us to determine (beside physical resource availability) how much resources can a tenant/reservation ask for in the Plan. This is both per-reservation and across reservation from a user (or group). We contributed so far a couple of simple policies allowing to enforce instantaneous and over-time limits (e.g., each user can grab up to 30% of the plan instantaneously, but no more than an average of 5% over a 24h period of time). Internally at MS, we are developing other policies that are specific to business-rules we care to enforce in our clusters. By design, creating a new SharingPolicy that match your business settings is fairly easy (narrow API and easy configuration mechanics). Since the Plan stores past (up to a window of time), present, future reservations, the policy can be very sophisticated, and explicit. Also given the run-lenght-encoded representation of the allocations, algos can be quite efficient. *ReservationAgent* The reservation agents are the core of the placement logic. We developed a few, which optimize for different things (e.g., minimize cost of the allocation by smoothing out the plan, or placing as late/early as possible in the window of feasibility). Again this is an area of possible enhancement, where business logic can kick in and choose to prioritize certain types of allocations. *Enforcement mechanics* Finally, in order to "enforce" this planned decisions, we use dynamically created and resized queues (each reservation can contain one or more jobs, thus the queue mechanism is useful to reuse). Note that [~acmurthy]'s comment was fairly technical, and related to this last point. He was proposing to leverage application priorities instead of queues as an enforcement mechanisms. Both are feasible, and have some pros and cons. Overall using queues allowed us to reuse some more of the mechanisms (e.g., rely on the preemption policy, and all of the advancement people are contributing there). > YARN Admission Control/Planner: enhancing the resource allocation model with > time. > -- > > Key: YARN-1051 > URL: https://issues.apache.org/jira/browse/YARN-1051 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, resourcemanager, scheduler >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.6.0 > > Attachments: YARN-1051-design.pdf, YARN-1051.1.patch, > YARN-1051.patch, curino_MSR-TR-2013-108.pdf, socc14-paper15.pdf, > techreport.pdf > > > In this umbrella JIRA we propose to extend the YARN RM to handle time > explicitly, allowing users to "reserve" capacity over time. This is an > important step towards SLAs, long-running services, workflows, and helps for > gang scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2885) Create AMRMProxy request interceptor for distributed scheduling decisions for queueable containers
[ https://issues.apache.org/jira/browse/YARN-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141190#comment-15141190 ] Carlo Curino commented on YARN-2885: Cool. Just to be explicit about it I am +1 on committing this to the branch. > Create AMRMProxy request interceptor for distributed scheduling decisions for > queueable containers > -- > > Key: YARN-2885 > URL: https://issues.apache.org/jira/browse/YARN-2885 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Arun Suresh > Attachments: YARN-2885-yarn-2877.001.patch, > YARN-2885-yarn-2877.002.patch, YARN-2885-yarn-2877.full-2.patch, > YARN-2885-yarn-2877.full-3.patch, YARN-2885-yarn-2877.full.patch, > YARN-2885-yarn-2877.v4.patch, YARN-2885-yarn-2877.v5.patch, > YARN-2885-yarn-2877.v6.patch, YARN-2885-yarn-2877.v7.patch, > YARN-2885-yarn-2877.v8.patch, YARN-2885_api_changes.patch > > > We propose to add a Local ResourceManager (LocalRM) to the NM in order to > support distributed scheduling decisions. > Architecturally we leverage the RMProxy, introduced in YARN-2884. > The LocalRM makes distributed decisions for queuable containers requests. > Guaranteed-start requests are still handled by the central RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4360) Improve GreedyReservationAgent to support "early" allocations, and performance improvements
[ https://issues.apache.org/jira/browse/YARN-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141244#comment-15141244 ] Hudson commented on YARN-4360: -- SUCCESS: Integrated in Hadoop-trunk-Commit #9275 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9275/]) YARN-4360. Improve GreedyReservationAgent to support "early" (arun suresh: rev 5cf5c41a895f5ab8bf6270089f8cfdea50573a97) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/ReservationSystemTestUtil.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/IterativePlanner.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/GreedyReservationAgent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/StageAllocatorGreedyRLE.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/TestGreedyReservationAgent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/planning/AlignedPlannerWithGreedy.java > Improve GreedyReservationAgent to support "early" allocations, and > performance improvements > > > Key: YARN-4360 > URL: https://issues.apache.org/jira/browse/YARN-4360 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-4360.2.patch, YARN-4360.3.patch, YARN-4360.5.patch, > YARN-4360.6.patch, YARN-4360.7.patch, YARN-4360.8.patch, YARN-4360.patch > > > The GreedyReservationAgent allocates "as late as possible". Per various > conversations, it seems useful to have a mirror behavior that allocates as > early as possible. Also in the process we leverage improvements from > YARN-4358, and implement an RLE-aware StageAllocatorGreedy(RLE), which > significantly speeds up allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4682) AMRM client to log when AMRM token updated
[ https://issues.apache.org/jira/browse/YARN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141162#comment-15141162 ] Prabhu Joseph commented on YARN-4682: - [~ste...@apache.org] Steve, do i need to checkout branch-2. The issue "No AMRMToken" happened on hadoop-2.4.1. So like you mentioned, the fix of YARN-3103 and YARN-2212 is missing there. I am doing a testing with the YARN-3103 fix, for every yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-secs, the AMRMToken gets updated. How to decrease the life time of a token, trying to simulate the issue again. > AMRM client to log when AMRM token updated > -- > > Key: YARN-4682 > URL: https://issues.apache.org/jira/browse/YARN-4682 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Affects Versions: 2.7.2 >Reporter: Steve Loughran > Attachments: YARN-4682.patch > > Original Estimate: 0.25h > Remaining Estimate: 0.25h > > There's no information right now as to when the AMRM token gets updated; if > something has gone wrong with the update, you can't tell when it last when > through. > fix: add a log statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4412) Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers
[
https://issues.apache.org/jira/browse/YARN-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141349#comment-15141349
]
Carlo Curino commented on YARN-4412:
Hi [~asuresh], I really like the direction of the patch. I had several
conversations with folks writing "smarter" applications, which are asking for
some "visibility" in what happens in the cluster.
If I understand the structure correctly, with what you do we could either:
# pass this extra info between RM and AMRMProxy, but stripping them out before
it reach the app (thus preserving existing behavior, and hiding potentially
sensitive info about the cluster load).
# have the AMRMProxy forward this info to a "smarter" app.
I think it is important to be able to *enforce* the above behaviors, i.e., a
sneaky AM should not be able to talk directly with the RM, pretending to be the
AMRMProxy and grabbing those extra fields. This could be accomplished with
an extra round of "tokens" that allow to talk the extended protocol vs just the
basic one. A trusted app can receive this tokens, while a untrusted app, will
not. The AMRMProxy is part of infrastructure, so should have this special
tokens.
Does this make sense?
Given the above demand from app writers, I think it would be nice to
*generalize* what you are doing. It would be nice to have a more general
purpose {{ClusterStatus}} object to be passed down. A specific instantiation of
which is your
{{QueuedContainersStatus}}, which specifically returns a top-k of queueing
behavior at nodes. Just to make an example, I can easily see a latency-critical
serving service trying to figure out where best to place its tasks, to ask for
information
about average CPU/NET/DISK utilization of all available nodes, before
requesting to run on a few that are (according to this service custom metrics)
the best fit. This shouldn't be too hard, I am just proposing a more general
wrapper object,
which can allows us later on to leverage this very same mechanism, for more
than what you guys do today. I think it would make it a very valuable service
to provide to apps writers, especially as we head towards more and more
services.
Nits:
* I think documentation on the various classes would help. e.g., you introduce
a {{DistributedSchedulingService}}, that from other discussions I understand is
useful, but just staring at the code is hard to get why we need all this.
* In {{ResourceManager}}, if you are factoring out all the "guts" of
SchedulerEventDispatcher, can't we simply move the class out? There is nothing
left in RM, other than a local rename?
* Can you clarify what happens in {{DistributedSchedulingService.getServer()}}
? The comment has double negations and I am not clear on what what a
reflectiveBlockingService does.
* {{registerApplicationMasterForDistributedScheduling}} assumes resources will
have only cpu/mem. This might change soon. Is there any better way to load this
info from configuration? It would be nice to have a
config.getResource("blah"), which takes care of this.
* I see tests for {{TopKNodeSelector}}, but for nothing else. Is this enough?
> Create ClusterMonitor to compute ordered list of preferred NMs for
> OPPORTUNITIC containers
> --
>
> Key: YARN-4412
> URL: https://issues.apache.org/jira/browse/YARN-4412
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-4412-yarn-2877.v1.patch,
> YARN-4412-yarn-2877.v2.patch
>
>
> Introduce a Cluster Monitor that aggregates load information from individual
> Node Managers and computes an ordered list of preferred Node managers to be
> used as target Nodes for OPPORTUNISTIC container allocations.
> This list can be pushed out to the Node Manager (specifically the AMRMProxy
> running on the Node) via the Allocate Response. This will be used to make
> local Scheduling decisions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4420) Add REST API for List Reservations
[ https://issues.apache.org/jira/browse/YARN-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141427#comment-15141427 ] Hudson commented on YARN-4420: -- FAILURE: Integrated in Hadoop-trunk-Commit #9277 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9277/]) YARN-4420. Add REST API for List Reservations (Sean Po via curino) (=: rev b706cbc1bc0ab3572c01676fe7365df21eda7ffa) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesReservation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ReservationInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ReservationDefinitionInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ResourceAllocationInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ReservationListInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ReservationRequestInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ReservationRequestsInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/submit-reservation.json * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ReservationIdInfo.java > Add REST API for List Reservations > -- > > Key: YARN-4420 > URL: https://issues.apache.org/jira/browse/YARN-4420 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po >Priority: Minor > Attachments: YARN-4420.v1.patch, YARN-4420.v2.patch, > YARN-4420.v3.patch, YARN-4420.v4.patch, YARN-4420.v5.patch, YARN-4420.v6.patch > > > This JIRA tracks changes to the REST APIs of the reservation system and > enables querying the reservation on which reservations exists by "time-range, > and reservation-id". > This task has a dependency on YARN-4340. > This task is related to YARN-4683. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4183) Enabling generic application history forces every job to get a timeline service delegation token
[ https://issues.apache.org/jira/browse/YARN-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141444#comment-15141444 ] Naganarasimha G R commented on YARN-4183: - Hi [~jeagles] & [~mitdesai], Any thoughts on the conclusion ? > Enabling generic application history forces every job to get a timeline > service delegation token > > > Key: YARN-4183 > URL: https://issues.apache.org/jira/browse/YARN-4183 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Mit Desai >Assignee: Naganarasimha G R > Attachments: YARN-4183.1.patch, YARN-4183.v1.001.patch, > YARN-4183.v1.002.patch > > > When enabling just the Generic History Server and not the timeline server, > the system metrics publisher will not publish the events to the timeline > store as it checks if the timeline server and system metrics publisher are > enabled before creating a timeline client. > To make it work, if the timeline service flag is turned on, it will force > every yarn application to get a delegation token. > Instead of checking if timeline service is enabled, we should be checking if > application history server is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4682) AMRM client to log when AMRM token updated
[
https://issues.apache.org/jira/browse/YARN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141304#comment-15141304
]
Steve Loughran commented on YARN-4682:
--
to get a patch we can put into Hadoop, yes, you need to check out trunk or
branch-2; that's where all patches go in, with some being cherry-picked earlier.
Moving off hadoop 2.4 to 2.6.1 should fix your problem, but having a log entry
should help anyway.
how to decrease the expiry time? that should be the key. Looking at the code,
it needs to be at least 3 times the value of
{{yarn.am.liveness-monitor.expiry-interval-ms}}. Make that nice and short
> AMRM client to log when AMRM token updated
> --
>
> Key: YARN-4682
> URL: https://issues.apache.org/jira/browse/YARN-4682
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
> Attachments: YARN-4682.patch
>
> Original Estimate: 0.25h
> Remaining Estimate: 0.25h
>
> There's no information right now as to when the AMRM token gets updated; if
> something has gone wrong with the update, you can't tell when it last when
> through.
> fix: add a log statement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4360) Improve GreedyReservationAgent to support "early" allocations, and performance improvements
[ https://issues.apache.org/jira/browse/YARN-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141355#comment-15141355 ] Carlo Curino commented on YARN-4360: [~asuresh], thanks so much for the reviews and commit. > Improve GreedyReservationAgent to support "early" allocations, and > performance improvements > > > Key: YARN-4360 > URL: https://issues.apache.org/jira/browse/YARN-4360 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-4360.2.patch, YARN-4360.3.patch, YARN-4360.5.patch, > YARN-4360.6.patch, YARN-4360.7.patch, YARN-4360.8.patch, YARN-4360.patch > > > The GreedyReservationAgent allocates "as late as possible". Per various > conversations, it seems useful to have a mirror behavior that allocates as > early as possible. Also in the process we leverage improvements from > YARN-4358, and implement an RLE-aware StageAllocatorGreedy(RLE), which > significantly speeds up allocation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4628) TopCli to support Application priority
[
https://issues.apache.org/jira/browse/YARN-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4628:
---
Attachment: 0003-YARN-4628.patch
[~sunilg]
Thanks for review comment.
By default if we are sorting on prority will be done in descending order since
{{ascendingSort = false}} by default . (Higher priority first) .
{noformat}
if (ascendingSort) {
Collections.sort(appsInfo, comparator);
} else {
Collections.sort(appsInfo, Collections.reverseOrder(comparator));
}
{noformat}
Also we do have option for reversing in in top cli.
[~vvasudev]
Handled latest review comments too. Attaching lated patch. Please do review
> TopCli to support Application priority
> ---
>
> Key: YARN-4628
> URL: https://issues.apache.org/jira/browse/YARN-4628
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Minor
> Attachments: 0001-YARN-4628.patch, 0002-YARN-4628.patch,
> 0003-YARN-4628.patch
>
>
> In Topcli support Apppriority
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4628) TopCli to support Application priority
[
https://issues.apache.org/jira/browse/YARN-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141661#comment-15141661
]
Hadoop QA commented on YARN-4628:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: patch
generated 1 new + 151 unchanged - 0 fixed = 152 total (was 151) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 38s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 48s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.yarn.client.TestGetGroups |
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.yarn.client.cli.TestYarnCLI |
| | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
| JDK v1.7.0_91 Failed junit tests | hadoop.yarn.client.TestGetGroups |
| JDK v1.7.0_91
[jira] [Updated] (YARN-2575) Consider creating separate ACLs for Reservation create/update/delete/list ops
[ https://issues.apache.org/jira/browse/YARN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-2575: -- Attachment: YARN-2575.v11.patch Thanks [~asuresh] for the comments. I added your comments to patch YARN-2575.v11.patch, and successfully tested the it on a single node cluster. I just wanted to point out that the remaining 3 checkstyle issues existed in the past, and are caused by methods being too long as well as having too many parameters in a method definition. > Consider creating separate ACLs for Reservation create/update/delete/list ops > - > > Key: YARN-2575 > URL: https://issues.apache.org/jira/browse/YARN-2575 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Subru Krishnan >Assignee: Sean Po > Attachments: YARN-2575.v1.patch, YARN-2575.v10.patch, > YARN-2575.v11.patch, YARN-2575.v2.1.patch, YARN-2575.v2.patch, > YARN-2575.v3.patch, YARN-2575.v4.patch, YARN-2575.v5.patch, > YARN-2575.v6.patch, YARN-2575.v7.patch, YARN-2575.v8.patch, YARN-2575.v9.patch > > > YARN-1051 introduces the ReservationSystem and in the current implementation > anyone who can submit applications can also submit reservations. This JIRA is > to evaluate creating separate ACLs for Reservation create/update/delete ops. > Depends on YARN-4340 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4686) MiniYARNCluster.start() returns before cluster is completely started
[
https://issues.apache.org/jira/browse/YARN-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141736#comment-15141736
]
Eric Badger commented on YARN-4686:
---
Moving to YARN project, since this affects the MiniYarnCluster.
> MiniYARNCluster.start() returns before cluster is completely started
>
>
> Key: YARN-4686
> URL: https://issues.apache.org/jira/browse/YARN-4686
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
> Attachments: MAPREDUCE-6507.001.patch
>
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (YARN-4686) MiniYARNCluster.start() returns before cluster is completely started
[
https://issues.apache.org/jira/browse/YARN-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger moved MAPREDUCE-6507 to YARN-4686:
--
Assignee: (was: Eric Badger)
Target Version/s: 2.7.3 (was: 2.7.3)
Component/s: (was: test)
test
Key: YARN-4686 (was: MAPREDUCE-6507)
Project: Hadoop YARN (was: Hadoop Map/Reduce)
> MiniYARNCluster.start() returns before cluster is completely started
>
>
> Key: YARN-4686
> URL: https://issues.apache.org/jira/browse/YARN-4686
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
> Attachments: MAPREDUCE-6507.001.patch
>
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[
https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141869#comment-15141869
]
Hadoop QA commented on YARN-4676:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} YARN-4676 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787357/graceful-decommission-v2.patch
|
| JIRA Issue | YARN-4676 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/10549/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Automatic and Asynchronous Decommissioning Nodes Status Tracking
>
>
> Key: YARN-4676
> URL: https://issues.apache.org/jira/browse/YARN-4676
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Daniel Zhi
>Assignee: Daniel Zhi
> Labels: features
> Fix For: 2.8.0
>
> Attachments: GracefulDecommissionYarnNode.pdf,
> graceful-decommission-v2.patch
>
>
> DecommissioningNodeWatcher inside ResourceTrackingService tracks
> DECOMMISSIONING nodes status automatically and asynchronously after
> client/admin made the graceful decommission request. It tracks
> DECOMMISSIONING nodes status to decide when, after all running containers on
> the node have completed, will be transitioned into DECOMMISSIONED state.
> NodesListManager detect and handle include and exclude list changes to kick
> out decommission or recommission as necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Zhi updated YARN-4676: - Attachment: graceful-decommission-v2.patch > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Fix For: 2.8.0 > > Attachments: GracefulDecommissionYarnNode.pdf, > graceful-decommission-v0.patch, graceful-decommission-v1.patch, > graceful-decommission-v2.patch > > > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4420) Add REST API for List Reservations
[ https://issues.apache.org/jira/browse/YARN-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141853#comment-15141853 ] Hudson commented on YARN-4420: -- FAILURE: Integrated in Hadoop-trunk-Commit #9278 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9278/]) YARN-4420. Amend adding modification to CHANGES.txt (=: rev cb3a10362d3ce09f5ca9baf28bb4ad6b1a20089a) * hadoop-yarn-project/CHANGES.txt > Add REST API for List Reservations > -- > > Key: YARN-4420 > URL: https://issues.apache.org/jira/browse/YARN-4420 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po >Priority: Minor > Attachments: YARN-4420.v1.patch, YARN-4420.v2.patch, > YARN-4420.v3.patch, YARN-4420.v4.patch, YARN-4420.v5.patch, YARN-4420.v6.patch > > > This JIRA tracks changes to the REST APIs of the reservation system and > enables querying the reservation on which reservations exists by "time-range, > and reservation-id". > This task has a dependency on YARN-4340. > This task is related to YARN-4683. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Zhi updated YARN-4676: - Attachment: (was: graceful-decommission-v1.patch) > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Fix For: 2.8.0 > > Attachments: GracefulDecommissionYarnNode.pdf, > graceful-decommission-v2.patch > > > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Zhi updated YARN-4676: - Attachment: (was: graceful-decommission-v0.patch) > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Fix For: 2.8.0 > > Attachments: GracefulDecommissionYarnNode.pdf, > graceful-decommission-v2.patch > > > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4420) Add REST API for List Reservations
[ https://issues.apache.org/jira/browse/YARN-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141862#comment-15141862 ] Carlo Curino commented on YARN-4420: Thanks [~seanpo03], I committed this to trunk, branch-2, and branch-2.8. > Add REST API for List Reservations > -- > > Key: YARN-4420 > URL: https://issues.apache.org/jira/browse/YARN-4420 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po >Priority: Minor > Attachments: YARN-4420.v1.patch, YARN-4420.v2.patch, > YARN-4420.v3.patch, YARN-4420.v4.patch, YARN-4420.v5.patch, YARN-4420.v6.patch > > > This JIRA tracks changes to the REST APIs of the reservation system and > enables querying the reservation on which reservations exists by "time-range, > and reservation-id". > This task has a dependency on YARN-4340. > This task is related to YARN-4683. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4138) Roll back container resource allocation after resource increase token expires
[ https://issues.apache.org/jira/browse/YARN-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142052#comment-15142052 ] sandflee commented on YARN-4138: looks good to me too, thanks [~mding] > Roll back container resource allocation after resource increase token expires > - > > Key: YARN-4138 > URL: https://issues.apache.org/jira/browse/YARN-4138 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: MENG DING >Assignee: MENG DING > Attachments: YARN-4138-YARN-1197.1.patch, > YARN-4138-YARN-1197.2.patch, YARN-4138.3.patch, YARN-4138.4.patch, > YARN-4138.5.patch, YARN-4138.6.patch, YARN-4138.7.patch > > > In YARN-1651, after container resource increase token expires, the running > container is killed. > This ticket will change the behavior such that when a container resource > increase token expires, the resource allocation of the container will be > reverted back to the value before the increase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4682) AMRM client to log when AMRM token updated
[ https://issues.apache.org/jira/browse/YARN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-4682: Attachment: YARN-4682.patch.1 > AMRM client to log when AMRM token updated > -- > > Key: YARN-4682 > URL: https://issues.apache.org/jira/browse/YARN-4682 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Affects Versions: 2.7.2 >Reporter: Steve Loughran > Attachments: YARN-4682.patch, YARN-4682.patch.1 > > Original Estimate: 0.25h > Remaining Estimate: 0.25h > > There's no information right now as to when the AMRM token gets updated; if > something has gone wrong with the update, you can't tell when it last when > through. > fix: add a log statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (YARN-4628) Display application priority in yarn top
[ https://issues.apache.org/jira/browse/YARN-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142308#comment-15142308 ] Varun Vasudev edited comment on YARN-4628 at 2/11/16 6:32 AM: -- +1. Committed to trunk and branch-2 with a minor modification - changed "#PRIOR" to "PRIOR". Thanks [~bibinchundatt]! was (Author: vvasudev): Committed to trunk and branch-2 with a minor modification - changed "#PRIOR" to "PRIOR". Thanks [~bibinchundatt]! > Display application priority in yarn top > > > Key: YARN-4628 > URL: https://issues.apache.org/jira/browse/YARN-4628 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Fix For: 2.9.0 > > Attachments: 0001-YARN-4628.patch, 0002-YARN-4628.patch, > 0003-YARN-4628.patch > > > In Topcli support Apppriority -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4682) AMRM client to log when AMRM token updated
[
https://issues.apache.org/jira/browse/YARN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142305#comment-15142305
]
Hadoop QA commented on YARN-4682:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} YARN-4682 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787425/YARN-4682.patch.1 |
| JIRA Issue | YARN-4682 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/10551/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AMRM client to log when AMRM token updated
> --
>
> Key: YARN-4682
> URL: https://issues.apache.org/jira/browse/YARN-4682
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
> Attachments: YARN-4682.patch, YARN-4682.patch.1
>
> Original Estimate: 0.25h
> Remaining Estimate: 0.25h
>
> There's no information right now as to when the AMRM token gets updated; if
> something has gone wrong with the update, you can't tell when it last when
> through.
> fix: add a log statement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4655) Log uncaught exceptions/errors in various thread pools in YARN
[ https://issues.apache.org/jira/browse/YARN-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142345#comment-15142345 ] Sidharta Seethana commented on YARN-4655: - Thanks, [~vvasudev]. > Log uncaught exceptions/errors in various thread pools in YARN > -- > > Key: YARN-4655 > URL: https://issues.apache.org/jira/browse/YARN-4655 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.9.0 > > Attachments: YARN-4655.001.patch, YARN-4655.002.patch > > > For a detailed description, please see HADOOP-12748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4628) Display application priority in yarn top
[ https://issues.apache.org/jira/browse/YARN-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4628: Summary: Display application priority in yarn top (was: TopCli to support Application priority ) > Display application priority in yarn top > > > Key: YARN-4628 > URL: https://issues.apache.org/jira/browse/YARN-4628 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Attachments: 0001-YARN-4628.patch, 0002-YARN-4628.patch, > 0003-YARN-4628.patch > > > In Topcli support Apppriority -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4682) AMRM client to log when AMRM token updated
[ https://issues.apache.org/jira/browse/YARN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142301#comment-15142301 ] Prabhu Joseph commented on YARN-4682: - git checkout branch-2 git diff > YARN-4682.patch.1 But I am not seeing any difference with the previous patch. > AMRM client to log when AMRM token updated > -- > > Key: YARN-4682 > URL: https://issues.apache.org/jira/browse/YARN-4682 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Affects Versions: 2.7.2 >Reporter: Steve Loughran > Attachments: YARN-4682.patch, YARN-4682.patch.1 > > Original Estimate: 0.25h > Remaining Estimate: 0.25h > > There's no information right now as to when the AMRM token gets updated; if > something has gone wrong with the update, you can't tell when it last when > through. > fix: add a log statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4655) Log uncaught exceptions/errors in various thread pools in YARN
[ https://issues.apache.org/jira/browse/YARN-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142324#comment-15142324 ] Hudson commented on YARN-4655: -- FAILURE: Integrated in Hadoop-trunk-Commit #9283 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9283/]) YARN-4655. Log uncaught exceptions/errors in various thread pools in (vvasudev: rev fa00d3e20560bee412b49e5792595749a247a8ab) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/CleanerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/loghandler/NonAggregatingLogHandler.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/server/services/RegistryAdminService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/sharedcache/SharedCacheUploadService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RequestHedgingRMFailoverProxyProvider.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java > Log uncaught exceptions/errors in various thread pools in YARN > -- > > Key: YARN-4655 > URL: https://issues.apache.org/jira/browse/YARN-4655 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.9.0 > > Attachments: YARN-4655.001.patch, YARN-4655.002.patch > > > For a detailed description, please see HADOOP-12748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4628) Display application priority in yarn top
[ https://issues.apache.org/jira/browse/YARN-4628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142317#comment-15142317 ] Hudson commented on YARN-4628: -- FAILURE: Integrated in Hadoop-trunk-Commit #9282 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9282/]) YARN-4628. Display application priority in yarn top. Contributed by (vvasudev: rev 663a80031cf6f17151622fff327171dedd976df3) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/TopCLI.java * hadoop-yarn-project/CHANGES.txt > Display application priority in yarn top > > > Key: YARN-4628 > URL: https://issues.apache.org/jira/browse/YARN-4628 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Minor > Fix For: 2.9.0 > > Attachments: 0001-YARN-4628.patch, 0002-YARN-4628.patch, > 0003-YARN-4628.patch > > > In Topcli support Apppriority -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2885) Create AMRMProxy request interceptor for distributed scheduling decisions for queueable containers
[ https://issues.apache.org/jira/browse/YARN-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142346#comment-15142346 ] Arun Suresh commented on YARN-2885: --- Thanks [~curino], [~leftnoteasy], [~kishorch], [~kkaranasos] and [~giovanni.fumarola] for all the reviews. I plan to commit this to branch yarn-2877 once I get a good jenkins to run against it > Create AMRMProxy request interceptor for distributed scheduling decisions for > queueable containers > -- > > Key: YARN-2885 > URL: https://issues.apache.org/jira/browse/YARN-2885 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Arun Suresh > Attachments: YARN-2885-yarn-2877.001.patch, > YARN-2885-yarn-2877.002.patch, YARN-2885-yarn-2877.full-2.patch, > YARN-2885-yarn-2877.full-3.patch, YARN-2885-yarn-2877.full.patch, > YARN-2885-yarn-2877.v4.patch, YARN-2885-yarn-2877.v5.patch, > YARN-2885-yarn-2877.v6.patch, YARN-2885-yarn-2877.v7.patch, > YARN-2885-yarn-2877.v8.patch, YARN-2885_api_changes.patch > > > We propose to add a Local ResourceManager (LocalRM) to the NM in order to > support distributed scheduling decisions. > Architecturally we leverage the RMProxy, introduced in YARN-2884. > The LocalRM makes distributed decisions for queuable containers requests. > Guaranteed-start requests are still handled by the central RM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4655) Log uncaught exceptions/errors in various thread pools in YARN
[ https://issues.apache.org/jira/browse/YARN-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-4655: Issue Type: Improvement (was: Bug) > Log uncaught exceptions/errors in various thread pools in YARN > -- > > Key: YARN-4655 > URL: https://issues.apache.org/jira/browse/YARN-4655 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Attachments: YARN-4655.001.patch, YARN-4655.002.patch > > > For a detailed description, please see HADOOP-12748 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[
https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142338#comment-15142338
]
Hadoop QA commented on YARN-4676:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 47s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 1s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 48s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 55s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 38s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 14s
{color} | {color:red} root: patch generated 138 new + 787 unchanged - 1 fixed =
925 total (was 788) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 143 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hadoop-common-project/hadoop-common generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 23s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 48s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 13s
{color} | {color:green}
[jira] [Updated] (YARN-4681) ProcfsBasedProcessTree should not calculate private clean pages
[
https://issues.apache.org/jira/browse/YARN-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Lukavsky updated YARN-4681:
---
Attachment: YARN-4681.patch
Added backward compatibility, removed some warnings.
> ProcfsBasedProcessTree should not calculate private clean pages
> ---
>
> Key: YARN-4681
> URL: https://issues.apache.org/jira/browse/YARN-4681
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Jan Lukavsky
> Attachments: YARN-4681.patch, YARN-4681.patch
>
>
> ProcfsBasedProcessTree in Node manager calculates memory used by a process
> tree by parsing {{/etc//smaps}}, where it calculates {{min(Pss,
> Shared_Dirty) + Private_Dirty + Private_Clean}}. Because not {{mlocked}}
> private clean pages can be reclaimed by kernel, this should be changed to
> calculating only {{Locked}} pages instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4685) AM blacklisting result in application to get hanged
[
https://issues.apache.org/jira/browse/YARN-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4685:
Summary: AM blacklisting result in application to get hanged (was: AM
blacklist addition/removal should get updated for every allocate call from
RMAppAttemptImpl.)
> AM blacklisting result in application to get hanged
> ---
>
> Key: YARN-4685
> URL: https://issues.apache.org/jira/browse/YARN-4685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>
> AM blacklist addition or removal is updated only when RMAppAttempt is
> scheduled i.e {{RMAppAttemptImpl#ScheduleTransition#transition}}. But once
> attempt is scheduled if there is any removeNode/addNode in cluster then this
> is not updated to {{BlackListManager#refreshNodeHostCount}}. This leads
> BlackListManager to operate on stale NM's count. And application is in
> ACCEPTED state and wait forever even if we add more nodes to cluster.
> Solution is update BlacklistManager for every
> {{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} call. This
> ensures if there is any addition/removal in nodes, this will be updated to
> BlacklistManager
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4685) AM blacklist addition/removal should get updated for every allocate call from RMAppAttemptImpl.
[
https://issues.apache.org/jira/browse/YARN-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4685:
Priority: Major (was: Blocker)
> AM blacklist addition/removal should get updated for every allocate call from
> RMAppAttemptImpl.
> ---
>
> Key: YARN-4685
> URL: https://issues.apache.org/jira/browse/YARN-4685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>
> AM blacklist addition or removal is updated only when RMAppAttempt is
> scheduled i.e {{RMAppAttemptImpl#ScheduleTransition#transition}}. But once
> attempt is scheduled if there is any removeNode/addNode in cluster then this
> is not updated to {{BlackListManager#refreshNodeHostCount}}. This leads
> BlackListManager to operate on stale NM's count. And application is in
> ACCEPTED state and wait forever even if we add more nodes to cluster.
> *Solution* is update BlacklistManager for every
> {{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} call. This
> ensures if there is any addition/removal in nodes, this will be updated to
> BlacklistManager
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4685) AM blacklist addition/removal should get updated for every allocate call from RMAppAttemptImpl.
[
https://issues.apache.org/jira/browse/YARN-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4685:
Description:
AM blacklist addition or removal is updated only when RMAppAttempt is scheduled
i.e {{RMAppAttemptImpl#ScheduleTransition#transition}}. But once attempt is
scheduled if there is any removeNode/addNode in cluster then this is not
updated to {{BlackListManager#refreshNodeHostCount}}. This leads
BlackListManager to operate on stale NM's count. And application is in ACCEPTED
state and wait forever even if we add more nodes to cluster.
Solution is update BlacklistManager for every
{{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} call. This
ensures if there is any addition/removal in nodes, this will be updated to
BlacklistManager
was:
AM blacklist addition or removal is updated only when RMAppAttempt is scheduled
i.e {{RMAppAttemptImpl#ScheduleTransition#transition}}. But once attempt is
scheduled if there is any removeNode/addNode in cluster then this is not
updated to {{BlackListManager#refreshNodeHostCount}}. This leads
BlackListManager to operate on stale NM's count. And application is in ACCEPTED
state and wait forever even if we add more nodes to cluster.
*Solution* is update BlacklistManager for every
{{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} call. This
ensures if there is any addition/removal in nodes, this will be updated to
BlacklistManager
> AM blacklist addition/removal should get updated for every allocate call from
> RMAppAttemptImpl.
> ---
>
> Key: YARN-4685
> URL: https://issues.apache.org/jira/browse/YARN-4685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>
> AM blacklist addition or removal is updated only when RMAppAttempt is
> scheduled i.e {{RMAppAttemptImpl#ScheduleTransition#transition}}. But once
> attempt is scheduled if there is any removeNode/addNode in cluster then this
> is not updated to {{BlackListManager#refreshNodeHostCount}}. This leads
> BlackListManager to operate on stale NM's count. And application is in
> ACCEPTED state and wait forever even if we add more nodes to cluster.
> Solution is update BlacklistManager for every
> {{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} call. This
> ensures if there is any addition/removal in nodes, this will be updated to
> BlacklistManager
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4684) TestYarnCLI#testGetContainers failing in trunk
[
https://issues.apache.org/jira/browse/YARN-4684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140508#comment-15140508
]
Bibin A Chundatt commented on YARN-4684:
All testcase failures are already tracked as part for YARN-4478
> TestYarnCLI#testGetContainers failing in trunk
> ---
>
> Key: YARN-4684
> URL: https://issues.apache.org/jira/browse/YARN-4684
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4684.patch
>
>
> TestYarnCLI#testGetContainers failing in CN locale
> {noformat}
> 2016-02-10 12:32:24,309 INFO mortbay.log (Slf4jLog.java:info(67)) - [Total
> number of containers :3
> Container-Id Start Time Finish
> Time StateHost Node Http Address
> LOG-URL
> container_1234_0005_01_01?? ??? 01 08:00:01 +0800 1970
> ?? ??? 01 08:00:05 +0800 1970 COMPLETE
> host:1234http://host:2345 logURL
> container_1234_0005_01_02?? ??? 01 08:00:01 +0800 1970
> ?? ??? 01 08:00:05 +0800 1970 COMPLETE
> host:1234http://host:2345 logURL
> container_1234_0005_01_03?? ??? 01 08:00:01 +0800 1970
> N/A RUNNING host:1234
> http://host:2345
> ]
> 2016-02-10 12:32:24,309 INFO mortbay.log (Slf4jLog.java:info(67)) -
> OutputFrom command
> 2016-02-10 12:32:24,309 INFO mortbay.log (Slf4jLog.java:info(67)) - [Total
> number of containers :3
> Container-Id Start Time Finish
> Time StateHost Node Http Address
> LOG-URL
> container_1234_0005_01_01鏄熸湡鍥? 涓?鏈? 01 08:00:01 +0800 1970
> 鏄熸湡鍥? 涓?鏈? 01 08:00:05 +0800 1970 COMPLETE
> host:1234http://host:2345 logURL
> container_1234_0005_01_02鏄熸湡鍥? 涓?鏈? 01 08:00:01 +0800 1970
> 鏄熸湡鍥? 涓?鏈? 01 08:00:05 +0800 1970 COMPLETE
> host:1234http://host:2345 logURL
> container_1234_0005_01_03鏄熸湡鍥? 涓?鏈? 01 08:00:01 +0800 1970
> N/A RUNNING host:1234
> http://host:2345
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4681) ProcfsBasedProcessTree should not calculate private clean pages
[
https://issues.apache.org/jira/browse/YARN-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140515#comment-15140515
]
Jan Lukavsky commented on YARN-4681:
[~cnauroth], I tested this patch against our jobs and it kind of helps, but
doesn't solve the whole problem. Another problem is that we see spikes of
direct memory allocations (so far I didn't track where exactly they come from),
but it lead me to a thought, that it might help not to calculate the exact
memory consumption of a container, but to average it over some time period
(configurable, default zero, which would lead to the current behavior).
So, first I will modify the patch as you suggest (so that if the Locked field
is missing, then the behavior of the ProcfsBasedProcessTree will be exactly the
same as before). I will then try to add the time averaging and let you know if
it helped.
Regarding the more aggresive strategies, I made some experiments and I don't
think it would help.
> ProcfsBasedProcessTree should not calculate private clean pages
> ---
>
> Key: YARN-4681
> URL: https://issues.apache.org/jira/browse/YARN-4681
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Jan Lukavsky
> Attachments: YARN-4681.patch
>
>
> ProcfsBasedProcessTree in Node manager calculates memory used by a process
> tree by parsing {{/etc//smaps}}, where it calculates {{min(Pss,
> Shared_Dirty) + Private_Dirty + Private_Clean}}. Because not {{mlocked}}
> private clean pages can be reclaimed by kernel, this should be changed to
> calculating only {{Locked}} pages instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4685) AM blacklist addition/removal should get updated for every allocate call from RMAppAttemptImpl.
[
https://issues.apache.org/jira/browse/YARN-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140512#comment-15140512
]
Rohith Sharma K S commented on YARN-4685:
-
Currently RMAppAttemptImpl start calling allocate method only when
CONTAINER_ALLOCATED event triggered. But If container is not allocated then
RMAppAttemptImpl will not call allocate method continuously. So even if we add
code for sending updated blacklist addition/removal in
{{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} that does not
useful. Need to think alternatives to handle this scenario
> AM blacklist addition/removal should get updated for every allocate call from
> RMAppAttemptImpl.
> ---
>
> Key: YARN-4685
> URL: https://issues.apache.org/jira/browse/YARN-4685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>
> AM blacklist addition or removal is updated only when RMAppAttempt is
> scheduled i.e {{RMAppAttemptImpl#ScheduleTransition#transition}}. But once
> attempt is scheduled if there is any removeNode/addNode in cluster then this
> is not updated to {{BlackListManager#refreshNodeHostCount}}. This leads
> BlackListManager to operate on stale NM's count. And application is in
> ACCEPTED state and wait forever even if we add more nodes to cluster.
> Solution is update BlacklistManager for every
> {{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} call. This
> ensures if there is any addition/removal in nodes, this will be updated to
> BlacklistManager
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4685) AM blacklist addition/removal should get updated for every allocate call from RMAppAttemptImpl.
[
https://issues.apache.org/jira/browse/YARN-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140526#comment-15140526
]
Rohith Sharma K S commented on YARN-4685:
-
One of the case where application got stuck is
# Cluster started with 2 Node initially and submitted 1 application.
# Attempt 1 is failed with disk failed in NM-1. Attempt-2 got created making
NM-1 as blacklisted node.
# NM-2 got removed from cluster. Only NM-1 is in cluster.
# Since NM-1 is blacklisted, no more containers are assigned to NM-1.
# In cluster only 1 node is there and that too blacklisted, so no more
container are assigning to NM-1 even after Node NM-1 is reconnected after
removing disk space.
> AM blacklist addition/removal should get updated for every allocate call from
> RMAppAttemptImpl.
> ---
>
> Key: YARN-4685
> URL: https://issues.apache.org/jira/browse/YARN-4685
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>
> AM blacklist addition or removal is updated only when RMAppAttempt is
> scheduled i.e {{RMAppAttemptImpl#ScheduleTransition#transition}}. But once
> attempt is scheduled if there is any removeNode/addNode in cluster then this
> is not updated to {{BlackListManager#refreshNodeHostCount}}. This leads
> BlackListManager to operate on stale NM's count. And application is in
> ACCEPTED state and wait forever even if we add more nodes to cluster.
> Solution is update BlacklistManager for every
> {{RMAppAttemptImpl#AMContainerAllocatedTransition#transition}} call. This
> ensures if there is any addition/removal in nodes, this will be updated to
> BlacklistManager
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4682) AMRM client to log when AMRM token updated
[ https://issues.apache.org/jira/browse/YARN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prabhu Joseph updated YARN-4682: Attachment: YARN-4682.patch > AMRM client to log when AMRM token updated > -- > > Key: YARN-4682 > URL: https://issues.apache.org/jira/browse/YARN-4682 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Affects Versions: 2.7.2 >Reporter: Steve Loughran > Attachments: YARN-4682.patch > > Original Estimate: 0.25h > Remaining Estimate: 0.25h > > There's no information right now as to when the AMRM token gets updated; if > something has gone wrong with the update, you can't tell when it last when > through. > fix: add a log statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4682) AMRM client to log when AMRM token updated
[ https://issues.apache.org/jira/browse/YARN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140720#comment-15140720 ] Prabhu Joseph commented on YARN-4682: - [~ste...@apache.org] Added a info log. > AMRM client to log when AMRM token updated > -- > > Key: YARN-4682 > URL: https://issues.apache.org/jira/browse/YARN-4682 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Affects Versions: 2.7.2 >Reporter: Steve Loughran > Attachments: YARN-4682.patch > > Original Estimate: 0.25h > Remaining Estimate: 0.25h > > There's no information right now as to when the AMRM token gets updated; if > something has gone wrong with the update, you can't tell when it last when > through. > fix: add a log statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4629) Distributed shell breaks under strong security
[ https://issues.apache.org/jira/browse/YARN-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140729#comment-15140729 ] Steve Loughran commented on YARN-4629: -- +1 I've already been testing a variant of this in slider; in a non-HA system it works without any compatibility problems. > Distributed shell breaks under strong security > -- > > Key: YARN-4629 > URL: https://issues.apache.org/jira/browse/YARN-4629 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell, security >Affects Versions: 2.7.1 > Environment: Secure cluster with the RM principal listed with a > /_HOST entry to be expanded, most common with YARN HA enabled. >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Attachments: YARN-4629.001.patch, YARN-4629.002.patch, > YARN-4629.003.patch, YARN-4629.004.patch > > > If the auth_to_local is set to map requests from unknown hosts to nobody, the > dist shell app fails. The reason is that the client doesn't translate the > _HOST placeholder to the local hostname. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4682) AMRM client to log when AMRM token updated
[
https://issues.apache.org/jira/browse/YARN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140732#comment-15140732
]
Hadoop QA commented on YARN-4682:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color}
| {color:red} YARN-4682 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12787249/YARN-4682.patch |
| JIRA Issue | YARN-4682 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/10544/console |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AMRM client to log when AMRM token updated
> --
>
> Key: YARN-4682
> URL: https://issues.apache.org/jira/browse/YARN-4682
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
> Attachments: YARN-4682.patch
>
> Original Estimate: 0.25h
> Remaining Estimate: 0.25h
>
> There's no information right now as to when the AMRM token gets updated; if
> something has gone wrong with the update, you can't tell when it last when
> through.
> fix: add a log statement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4653) Document YARN security model from the perspective of Application Developers
[ https://issues.apache.org/jira/browse/YARN-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140742#comment-15140742 ] Steve Loughran commented on YARN-4653: -- ok, to confirm then # the token handed off by the RM to the NM to localize is refreshed/updated as needed. # no tokens in the app launch context are refreshed. That is, if it has an out of date hdfs token —that token is not renewed # therefore, to survive AM restart after token failure, your AM has to get the NMs to localize the keytab or make no HDFS accesses until (somehow) a new token has been passed to them from a client. This is what I will say > Document YARN security model from the perspective of Application Developers > --- > > Key: YARN-4653 > URL: https://issues.apache.org/jira/browse/YARN-4653 > Project: Hadoop YARN > Issue Type: Task > Components: site >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: YARN-4653-001.patch, YARN-4653-002.patch, > YARN-4653-003.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > What YARN apps need to do for security today is generally copied direct from > distributed shell, with a bit of [ill-informed > superstition|https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/yarn.html] > being the sole prose. > We need a normative document in the YARN site covering > # the needs for YARN security > # token creation for AM launch > # how the RM gets involved > # token propagation on container launch > # token renewal strategies > # How to get tokens for other apps like HBase and Hive. > # how to work under OOzie > Perhaps the WritingYarnApplications.md doc is updated, otherwise why not just > link to the relevant bit of the distributed shell client on github for a > guarantee of staying up to date? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4629) Distributed shell breaks under strong security
[ https://issues.apache.org/jira/browse/YARN-4629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140747#comment-15140747 ] Hudson commented on YARN-4629: -- FAILURE: Integrated in Hadoop-trunk-Commit #9272 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9272/]) YARN-4629. Distributed shell breaks under strong security. (Daniel (stevel: rev e9a622606f69dc926a950d4dd61fe3f16f378509) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/util/YarnClientUtils.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/util/TestYarnClientUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/util/package-info.java > Distributed shell breaks under strong security > -- > > Key: YARN-4629 > URL: https://issues.apache.org/jira/browse/YARN-4629 > Project: Hadoop YARN > Issue Type: Bug > Components: applications/distributed-shell, security >Affects Versions: 2.7.1 > Environment: Secure cluster with the RM principal listed with a > /_HOST entry to be expanded, most common with YARN HA enabled. >Reporter: Daniel Templeton >Assignee: Daniel Templeton > Fix For: 2.9.0 > > Attachments: YARN-4629.001.patch, YARN-4629.002.patch, > YARN-4629.003.patch, YARN-4629.004.patch > > > If the auth_to_local is set to map requests from unknown hosts to nobody, the > dist shell app fails. The reason is that the client doesn't translate the > _HOST placeholder to the local hostname. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4682) AMRM client to log when AMRM token updated
[ https://issues.apache.org/jira/browse/YARN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15140754#comment-15140754 ] Steve Loughran commented on YARN-4682: -- Prabhu, the patch isn't applying as it's against an old version of Hadoop. 2.6.0 probably # can you check out trunk/branch-2 and create the patch there # the version of code you are running doesn't have YARN-3103 applied (which is is why the patch is failing). That's probably the root cause of the token renewal problems discussed on the mailing list. If you are using hadoop 2.6.0, you need to upgrade to 2.6.1. If you are using something else (EMR, CDH, HDP, ...) , can you add it and its version to the environment field, so we can track how those releases are in relation to this issue. > AMRM client to log when AMRM token updated > -- > > Key: YARN-4682 > URL: https://issues.apache.org/jira/browse/YARN-4682 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Affects Versions: 2.7.2 >Reporter: Steve Loughran > Attachments: YARN-4682.patch > > Original Estimate: 0.25h > Remaining Estimate: 0.25h > > There's no information right now as to when the AMRM token gets updated; if > something has gone wrong with the update, you can't tell when it last when > through. > fix: add a log statement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4519) potential deadlock of CapacityScheduler between decrease container and assign containers
[ https://issues.apache.org/jira/browse/YARN-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-4519: -- Fix Version/s: (was: 2.9.0) 2.8.0 > potential deadlock of CapacityScheduler between decrease container and assign > containers > > > Key: YARN-4519 > URL: https://issues.apache.org/jira/browse/YARN-4519 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: sandflee >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-4519.1.patch, YARN-4519.2.patch, YARN-4519.3.patch > > > In CapacityScheduler.allocate() , first get FiCaSchedulerApp sync lock, and > may be get CapacityScheduler's sync lock in decreaseContainer() > In scheduler thread, first get CapacityScheduler's sync lock in > allocateContainersToNode(), and may get FiCaSchedulerApp sync lock in > FicaSchedulerApp.assignContainers(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4519) potential deadlock of CapacityScheduler between decrease container and assign containers
[ https://issues.apache.org/jira/browse/YARN-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142158#comment-15142158 ] Jian He commented on YARN-4519: --- committed in 2.8 too > potential deadlock of CapacityScheduler between decrease container and assign > containers > > > Key: YARN-4519 > URL: https://issues.apache.org/jira/browse/YARN-4519 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: sandflee >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-4519.1.patch, YARN-4519.2.patch, YARN-4519.3.patch > > > In CapacityScheduler.allocate() , first get FiCaSchedulerApp sync lock, and > may be get CapacityScheduler's sync lock in decreaseContainer() > In scheduler thread, first get CapacityScheduler's sync lock in > allocateContainersToNode(), and may get FiCaSchedulerApp sync lock in > FicaSchedulerApp.assignContainers(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4519) potential deadlock of CapacityScheduler between decrease container and assign containers
[ https://issues.apache.org/jira/browse/YARN-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-4519: -- Target Version/s: 2.8.0 (was: 2.9.0) > potential deadlock of CapacityScheduler between decrease container and assign > containers > > > Key: YARN-4519 > URL: https://issues.apache.org/jira/browse/YARN-4519 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: sandflee >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-4519.1.patch, YARN-4519.2.patch, YARN-4519.3.patch > > > In CapacityScheduler.allocate() , first get FiCaSchedulerApp sync lock, and > may be get CapacityScheduler's sync lock in decreaseContainer() > In scheduler thread, first get CapacityScheduler's sync lock in > allocateContainersToNode(), and may get FiCaSchedulerApp sync lock in > FicaSchedulerApp.assignContainers(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4138) Roll back container resource allocation after resource increase token expires
[ https://issues.apache.org/jira/browse/YARN-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142161#comment-15142161 ] Hudson commented on YARN-4138: -- SUCCESS: Integrated in Hadoop-trunk-Commit #9280 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9280/]) YARN-4138. Roll back container resource allocation after resource (jianhe: rev d16b17b4d299b4d58f879a2a15708bacd0938685) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/AllocationExpirationInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/ContainerAllocationExpirer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerNMDoneChangeResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/ContainerExpiredSchedulerEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestIncreaseAllocationExpirer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java > Roll back container resource allocation after resource increase token expires > - > > Key: YARN-4138 > URL: https://issues.apache.org/jira/browse/YARN-4138 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, nodemanager, resourcemanager >Reporter: MENG DING >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-4138-YARN-1197.1.patch, > YARN-4138-YARN-1197.2.patch, YARN-4138.3.patch, YARN-4138.4.patch, > YARN-4138.5.patch, YARN-4138.6.patch, YARN-4138.7.patch > > > In YARN-1651, after container resource increase token expires, the running > container is killed. > This ticket will change the behavior such that when a container resource > increase token expires, the resource allocation of the container will be > reverted back to the value before the increase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4519) potential deadlock of CapacityScheduler between decrease container and assign containers
[ https://issues.apache.org/jira/browse/YARN-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142162#comment-15142162 ] Hudson commented on YARN-4519: -- SUCCESS: Integrated in Hadoop-trunk-Commit #9280 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9280/]) Move YARN-4519 in CHANGES.txt to 2.8 (jianhe: rev 39a71b606a4abdc46ea6dfce2b79480b47ec18b5) * hadoop-yarn-project/CHANGES.txt > potential deadlock of CapacityScheduler between decrease container and assign > containers > > > Key: YARN-4519 > URL: https://issues.apache.org/jira/browse/YARN-4519 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: sandflee >Assignee: MENG DING > Fix For: 2.8.0 > > Attachments: YARN-4519.1.patch, YARN-4519.2.patch, YARN-4519.3.patch > > > In CapacityScheduler.allocate() , first get FiCaSchedulerApp sync lock, and > may be get CapacityScheduler's sync lock in decreaseContainer() > In scheduler thread, first get CapacityScheduler's sync lock in > allocateContainersToNode(), and may get FiCaSchedulerApp sync lock in > FicaSchedulerApp.assignContainers(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3223) Resource update during NM graceful decommission
[ https://issues.apache.org/jira/browse/YARN-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142170#comment-15142170 ] Brook Zhou commented on YARN-3223: -- Hi [~djp], can you please review? > Resource update during NM graceful decommission > --- > > Key: YARN-3223 > URL: https://issues.apache.org/jira/browse/YARN-3223 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, resourcemanager >Affects Versions: 2.7.1 >Reporter: Junping Du >Assignee: Brook Zhou > Attachments: YARN-3223-v0.patch, YARN-3223-v1.patch, > YARN-3223-v2.patch, YARN-3223-v3.patch, YARN-3223-v4.patch, YARN-3223-v5.patch > > > During NM graceful decommission, we should handle resource update properly, > include: make RMNode keep track of old resource for possible rollback, keep > available resource to 0 and used resource get updated when > container finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2575) Consider creating separate ACLs for Reservation create/update/delete/list ops
[ https://issues.apache.org/jira/browse/YARN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141920#comment-15141920 ] Sean Po commented on YARN-2575: --- Created a JIRA for documentation of Reservation ACLs. > Consider creating separate ACLs for Reservation create/update/delete/list ops > - > > Key: YARN-2575 > URL: https://issues.apache.org/jira/browse/YARN-2575 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Subru Krishnan >Assignee: Sean Po > Attachments: YARN-2575.v1.patch, YARN-2575.v10.patch, > YARN-2575.v11.patch, YARN-2575.v2.1.patch, YARN-2575.v2.patch, > YARN-2575.v3.patch, YARN-2575.v4.patch, YARN-2575.v5.patch, > YARN-2575.v6.patch, YARN-2575.v7.patch, YARN-2575.v8.patch, YARN-2575.v9.patch > > > YARN-1051 introduces the ReservationSystem and in the current implementation > anyone who can submit applications can also submit reservations. This JIRA is > to evaluate creating separate ACLs for Reservation create/update/delete ops. > Depends on YARN-4340 > Is related to YARN-4687 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4686) MiniYARNCluster.start() returns before cluster is completely started
[
https://issues.apache.org/jira/browse/YARN-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141939#comment-15141939
]
Hadoop QA commented on YARN-4686:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 44s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 39s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests |
hadoop.yarn.server.TestMiniYARNClusterForHA |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
| | hadoop.yarn.server.TestContainerManagerSecurity |
| JDK v1.7.0_95 Failed junit tests |
hadoop.yarn.server.TestMiniYARNClusterForHA |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
| | hadoop.yarn.server.TestContainerManagerSecurity |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785871/MAPREDUCE-6507.001.patch
|
| JIRA Issue | YARN-4686 |
| Optional Tests | asflicense
[jira] [Commented] (YARN-2575) Consider creating separate ACLs for Reservation create/update/delete/list ops
[ https://issues.apache.org/jira/browse/YARN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141947#comment-15141947 ] Sean Po commented on YARN-2575: --- YARN-4687 > Consider creating separate ACLs for Reservation create/update/delete/list ops > - > > Key: YARN-2575 > URL: https://issues.apache.org/jira/browse/YARN-2575 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Subru Krishnan >Assignee: Sean Po > Attachments: YARN-2575.v1.patch, YARN-2575.v10.patch, > YARN-2575.v11.patch, YARN-2575.v2.1.patch, YARN-2575.v2.patch, > YARN-2575.v3.patch, YARN-2575.v4.patch, YARN-2575.v5.patch, > YARN-2575.v6.patch, YARN-2575.v7.patch, YARN-2575.v8.patch, YARN-2575.v9.patch > > > YARN-1051 introduces the ReservationSystem and in the current implementation > anyone who can submit applications can also submit reservations. This JIRA is > to evaluate creating separate ACLs for Reservation create/update/delete ops. > Depends on YARN-4340 > Is related to YARN-4687 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2575) Consider creating separate ACLs for Reservation create/update/delete/list ops
[ https://issues.apache.org/jira/browse/YARN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-2575: - Description: YARN-1051 introduces the ReservationSystem and in the current implementation anyone who can submit applications can also submit reservations. This JIRA is to evaluate creating separate ACLs for Reservation create/update/delete ops. (was: YARN-1051 introduces the ReservationSystem and in the current implementation anyone who can submit applications can also submit reservations. This JIRA is to evaluate creating separate ACLs for Reservation create/update/delete ops. Depends on YARN-4340 Is related to YARN-4687) > Consider creating separate ACLs for Reservation create/update/delete/list ops > - > > Key: YARN-2575 > URL: https://issues.apache.org/jira/browse/YARN-2575 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Subru Krishnan >Assignee: Sean Po > Attachments: YARN-2575.v1.patch, YARN-2575.v10.patch, > YARN-2575.v11.patch, YARN-2575.v2.1.patch, YARN-2575.v2.patch, > YARN-2575.v3.patch, YARN-2575.v4.patch, YARN-2575.v5.patch, > YARN-2575.v6.patch, YARN-2575.v7.patch, YARN-2575.v8.patch, YARN-2575.v9.patch > > > YARN-1051 introduces the ReservationSystem and in the current implementation > anyone who can submit applications can also submit reservations. This JIRA is > to evaluate creating separate ACLs for Reservation create/update/delete ops. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Zhi updated YARN-4676: - Attachment: HADOOP-4676.003.patch > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Fix For: 2.8.0 > > Attachments: GracefulDecommissionYarnNode.pdf, HADOOP-4676.003.patch > > > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Zhi updated YARN-4676: - Attachment: (was: graceful-decommission-v2.patch) > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Fix For: 2.8.0 > > Attachments: GracefulDecommissionYarnNode.pdf > > > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4420) Add REST API for List Reservations
[ https://issues.apache.org/jira/browse/YARN-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141908#comment-15141908 ] Subru Krishnan commented on YARN-4420: -- Thanks [~seanpo03] for contributing the patch and [~curino] for reviewing/committing. > Add REST API for List Reservations > -- > > Key: YARN-4420 > URL: https://issues.apache.org/jira/browse/YARN-4420 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4420.v1.patch, YARN-4420.v2.patch, > YARN-4420.v3.patch, YARN-4420.v4.patch, YARN-4420.v5.patch, YARN-4420.v6.patch > > > This JIRA tracks changes to the REST APIs of the reservation system and > enables querying the reservation on which reservations exists by "time-range, > and reservation-id". > This task has a dependency on YARN-4340. > This task is related to YARN-4683. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4468) Document the general ReservationSystem functionality, and the REST API
[ https://issues.apache.org/jira/browse/YARN-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141909#comment-15141909 ] Carlo Curino commented on YARN-4468: [~subru] volunteered, to include in this patch the configuration in FairScheduler and CapacityScheduler. Waiting for that. > Document the general ReservationSystem functionality, and the REST API > -- > > Key: YARN-4468 > URL: https://issues.apache.org/jira/browse/YARN-4468 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-4468.1.patch, YARN-4468.rest-only.patch > > > This JIRA tracks effort to document the ReservationSystem functionality, and > the REST API access to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4687) Document Reservation ACLs
Sean Po created YARN-4687: - Summary: Document Reservation ACLs Key: YARN-4687 URL: https://issues.apache.org/jira/browse/YARN-4687 Project: Hadoop YARN Issue Type: Task Reporter: Sean Po Assignee: Sean Po Priority: Minor Related to YARN-2575 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2575) Consider creating separate ACLs for Reservation create/update/delete/list ops
[ https://issues.apache.org/jira/browse/YARN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Po updated YARN-2575: -- Description: YARN-1051 introduces the ReservationSystem and in the current implementation anyone who can submit applications can also submit reservations. This JIRA is to evaluate creating separate ACLs for Reservation create/update/delete ops. Depends on YARN-4340 Is related to YARN-4687 was: YARN-1051 introduces the ReservationSystem and in the current implementation anyone who can submit applications can also submit reservations. This JIRA is to evaluate creating separate ACLs for Reservation create/update/delete ops. Depends on YARN-4340 > Consider creating separate ACLs for Reservation create/update/delete/list ops > - > > Key: YARN-2575 > URL: https://issues.apache.org/jira/browse/YARN-2575 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, resourcemanager >Reporter: Subru Krishnan >Assignee: Sean Po > Attachments: YARN-2575.v1.patch, YARN-2575.v10.patch, > YARN-2575.v11.patch, YARN-2575.v2.1.patch, YARN-2575.v2.patch, > YARN-2575.v3.patch, YARN-2575.v4.patch, YARN-2575.v5.patch, > YARN-2575.v6.patch, YARN-2575.v7.patch, YARN-2575.v8.patch, YARN-2575.v9.patch > > > YARN-1051 introduces the ReservationSystem and in the current implementation > anyone who can submit applications can also submit reservations. This JIRA is > to evaluate creating separate ACLs for Reservation create/update/delete ops. > Depends on YARN-4340 > Is related to YARN-4687 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4687) Document Reservation ACLs
[ https://issues.apache.org/jira/browse/YARN-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-4687: - Issue Type: Sub-task (was: Task) Parent: YARN-2572 > Document Reservation ACLs > - > > Key: YARN-4687 > URL: https://issues.apache.org/jira/browse/YARN-4687 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sean Po >Assignee: Sean Po >Priority: Minor > > Related to YARN-2575 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2575) Consider creating separate ACLs for Reservation create/update/delete/list ops
[
https://issues.apache.org/jira/browse/YARN-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15141948#comment-15141948
]
Hadoop QA commented on YARN-2575:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 51s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} trunk passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 6s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 45s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 3 new +
369 unchanged - 3 fixed = 372 total (was 372) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 36s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 43s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 39s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 12s
{color}
[jira] [Updated] (YARN-4420) Add REST API for List Reservations
[ https://issues.apache.org/jira/browse/YARN-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-4420: - Description: This JIRA tracks changes to the REST APIs of the reservation system and enables querying the reservation on which reservations exists by "time-range, and reservation-id". (was: This JIRA tracks changes to the REST APIs of the reservation system and enables querying the reservation on which reservations exists by "time-range, and reservation-id". This task has a dependency on YARN-4340. This task is related to YARN-4683.) > Add REST API for List Reservations > -- > > Key: YARN-4420 > URL: https://issues.apache.org/jira/browse/YARN-4420 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, fairscheduler, resourcemanager >Reporter: Sean Po >Assignee: Sean Po >Priority: Minor > Fix For: 2.8.0 > > Attachments: YARN-4420.v1.patch, YARN-4420.v2.patch, > YARN-4420.v3.patch, YARN-4420.v4.patch, YARN-4420.v5.patch, YARN-4420.v6.patch > > > This JIRA tracks changes to the REST APIs of the reservation system and > enables querying the reservation on which reservations exists by "time-range, > and reservation-id". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4683) Document the List Reservations REST API
[ https://issues.apache.org/jira/browse/YARN-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-4683: - Description: YARN-4420 adds a REST API to list existing reservations in the Plan. This JIRA proposes adding documentation for the REST API (was: Related to YARN-4420) > Document the List Reservations REST API > --- > > Key: YARN-4683 > URL: https://issues.apache.org/jira/browse/YARN-4683 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sean Po >Assignee: Sean Po > > YARN-4420 adds a REST API to list existing reservations in the Plan. This > JIRA proposes adding documentation for the REST API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
