[jira] [Commented] (YARN-5889) Improve user-limit calculation in capacity scheduler
[
https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853626#comment-15853626
]
Sunil G commented on YARN-5889:
---
HI [~eepayne].
Sorry for troubling you again. I ran many similar test cases in my local
cluster. I could not get the case mentioned by you, even though i tested for
many time today and yesterday. So i thought I might be missing something.
I will try share my test case results here:
- I used {{YARN-5889.0010.patch}} patch for my tests.
- One node cluster (8GB).
- I used sleep job to make sure i have a control on test time and on number of
containers.
- AM and other containers were using 2G containers always.
*Test Scenario* (For 3 test cases, same scenario were used, only change is in
configuration)
1) Ran 4 containers including AM. Pending RR is 2Gb for one more container
2) Killed all non-AM containers 3 times (totally 9 containers were killed using
FORCEFUL_SHUTDOWN signal)
3) Killed AM containers once and job re-ran.
+Scenario 1:+ (One Queue 100%)
MULP = 100, ULFactor = 1
Output:
Application retook whole cluster always when one/more container were killed.
+Scenario 2:+ (One Queue 100%)
MULP = 50, ULFactor = 1
Output:
Application retook whole cluster always when one/more container were killed. UL
was 50%, still it got whole cluster.
+Scenario 3:+ (Two Queues 50% each)
MULP = 50, ULFactor = 2
Output:
Application retook whole cluster always when one/more container were killed. UL
was 50%, still it got whole cluster as UL factor was 2. (If its 1, then only
6Gb could be used).
Could you please help to take a look and share me what I might have missed from
your test scenario. I ll continue test some more scenarios and will keep posted.
> Improve user-limit calculation in capacity scheduler
>
>
> Key: YARN-5889
> URL: https://issues.apache.org/jira/browse/YARN-5889
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-5889.0001.patch,
> YARN-5889.0001.suggested.patchnotes, YARN-5889.0002.patch,
> YARN-5889.0003.patch, YARN-5889.0004.patch, YARN-5889.0005.patch,
> YARN-5889.0006.patch, YARN-5889.0007.patch, YARN-5889.0008.patch,
> YARN-5889.0009.patch, YARN-5889.0010.patch, YARN-5889.v0.patch,
> YARN-5889.v1.patch, YARN-5889.v2.patch
>
>
> Currently user-limit is computed during every heartbeat allocation cycle with
> a write lock. To improve performance, this tickets is focussing on moving
> user-limit calculation out of heartbeat allocation flow.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6027) Improve /flows API for more flexible filters fromid, collapse, userid
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-6027: Summary: Improve /flows API for more flexible filters fromid, collapse, userid (was: Support fromId for flows API ) > Improve /flows API for more flexible filters fromid, collapse, userid > - > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6146) Add Builder methods for TimelineEntityFilters
Rohith Sharma K S created YARN-6146: --- Summary: Add Builder methods for TimelineEntityFilters Key: YARN-6146 URL: https://issues.apache.org/jira/browse/YARN-6146 Project: Hadoop YARN Issue Type: Sub-task Reporter: Rohith Sharma K S The timeline filters are evolving and can be add more and more filters. It is better to start using Builder methods rather than changing constructor every time for adding new filters. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5893) Experiment with using PriorityQueue for storing starved applications
[ https://issues.apache.org/jira/browse/YARN-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853482#comment-15853482 ] Yufei Gu commented on YARN-5893: A PriorityBlockingQueue has been used in FSStarvedApps. Close this JIRA. > Experiment with using PriorityQueue for storing starved applications > > > Key: YARN-5893 > URL: https://issues.apache.org/jira/browse/YARN-5893 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Yufei Gu > > FSStarvedApps currently uses a BlockingQueue to store starved applications. > This leads to storing the apps in the order they were processed instead of > their priorities. Using a PriorityQueue will ensure we preempt for the > highest priority apps first. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5893) Experiment with using PriorityQueue for storing starved applications
[ https://issues.apache.org/jira/browse/YARN-5893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu resolved YARN-5893. Resolution: Won't Fix > Experiment with using PriorityQueue for storing starved applications > > > Key: YARN-5893 > URL: https://issues.apache.org/jira/browse/YARN-5893 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Yufei Gu > > FSStarvedApps currently uses a BlockingQueue to store starved applications. > This leads to storing the apps in the order they were processed instead of > their priorities. Using a PriorityQueue will ensure we preempt for the > highest priority apps first. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5271) ATS client doesn't work with Jersey 2 on the classpath
[ https://issues.apache.org/jira/browse/YARN-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853454#comment-15853454 ] Weiwei Yang commented on YARN-5271: --- Hello [~gtCarrera9] Thank you for watching on this. This is a handler for an abnormal condition that client uses a conflict jersey dependency against Yarn, so I thought disabling timeline service was better than fail. I'd love to hear if you or anyone else have any other good suggestions. > ATS client doesn't work with Jersey 2 on the classpath > -- > > Key: YARN-5271 > URL: https://issues.apache.org/jira/browse/YARN-5271 > Project: Hadoop YARN > Issue Type: Bug > Components: client, timelineserver >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Weiwei Yang > Labels: oct16-medium > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: YARN-5271.01.patch, YARN-5271.02.patch, > YARN-5271.branch-2.01.patch, YARN-5271-branch-2.8.01.patch > > > see SPARK-15343 : once Jersey 2 is on the CP, you can't instantiate a > timeline client, *even if the server is an ATS1.5 server and publishing is > via the FS* -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5703) ReservationAgents are not correctly configured
[ https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-5703: --- Attachment: YARN-5703.003.patch > ReservationAgents are not correctly configured > -- > > Key: YARN-5703 > URL: https://issues.apache.org/jira/browse/YARN-5703 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Po >Assignee: Manikandan R > Attachments: YARN-5703.001.patch, YARN-5703.002.patch, > YARN-5703.003.patch > > > In AbstractReservationSystem, the method that instantiates a ReservationAgent > does not properly initialize it with the appropriate configuration because it > expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured
[
https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853216#comment-15853216
]
Manikandan R commented on YARN-5703:
Thanks [~Naganarasimha] for review. I've attached newer patch based on comments
and given explanation for each one below.
* Ok, I used {code} agentclass = Configuration.getClass(String, Class, Class) {code} to create objects.
* I've defined few constants in {code}ReservationSchedulerConfiguration{code}.
To access those values through its corresponding getters method in
{code}init(ReservationSchedulerConfiguration conf) {code} of
AlignedPlannerWithGreedy & GreedyReservationAgent classes, I've defined the
method {code} void init(ReservationSchedulerConfiguration conf); {code} in
ReservationAgent.java. If I don't specify right sub class, those constants and
its getters won't be available? Do you see any issues here?
* Yes, configurable doesn't suit here. But,
{code}init(ReservationSchedulerConfiguration conf){code} has been implemented
already.
* Yes, {code}yarnConf{code} doesn't add value here. The reason for including
{code} init(ReservationSchedulerConfiguration conf) {code} in
IterativePlanner.java is, it extends {code}PlanningAlgorithm{code}, which again
has implemented {code}ReservationAgent{code}. Since configuration is available
in AlignedPlannerWithGreedy & GreedyReservationAgent classes, thought of
passing the same to IterativePlanner.java & TryManyReservationAgents.java as
both classes has implemented {code}ReservationAgent{code} interface, but there
is no use of it currently.
> ReservationAgents are not correctly configured
> --
>
> Key: YARN-5703
> URL: https://issues.apache.org/jira/browse/YARN-5703
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, resourcemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Sean Po
>Assignee: Manikandan R
> Attachments: YARN-5703.001.patch, YARN-5703.002.patch,
> YARN-5703.003.patch
>
>
> In AbstractReservationSystem, the method that instantiates a ReservationAgent
> does not properly initialize it with the appropriate configuration because it
> expects the ReservationAgent to implement Configurable.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
