Hi Vinod and Karthik
YARN-3849 was trying fix an irregularity in getting used queue resource for
ProportionalPremptionPolicy. Its not very complicated patch. And it fixes a
critical issue in preemption policy when used with DRC.
In PCPP, earlier to get the used capacity, total resource were multiplied
with absoluteCapacity per queue. Instead, we could get this resource
information direct with an api
{{curQueue.getQueueResourceUsage().getUsed(partition)}}
w/o using any more extra calculation. While using DRC we found that if we
do totalResource*absoluteCapacity, it may normalize the total resource
(either memory or vcore). To avoid, we could directly get the used capacity
per-queue per partition.
Rest all patch is to have DRC to support in test code of
ProportionalPremptionPolicy.
+ Wangda. Kindly share your thoughts.
Thank You
Sunil
On Thu, Nov 12, 2015 at 10:51 PM Karthik Kambatla <ka...@cloudera.com>
wrote:
> If YARN-3849 is a complicated patch, are we comfortable including it in
> 2.7.3? If it is not de-stabilizing, I am fine with it. Otherwise, it might
> make more sense to not include it in later 2.7.x.
>
> With my Cloudera hat on, inclusion/exclusion in 2.7.3 doesn't really
> matter.
>
> With my Apache hat on, the only way we inspire confidence to our users is
> increasingly stable maintenance releases. I know I have been harping on
> this a lot.
>
> On Tue, Nov 3, 2015 at 11:37 AM, Vinod Vavilapalli <
> vino...@hortonworks.com>
> wrote:
>
> > YARN-3849 looks like a bit of a non-trivial change this late for 2.7.2, I
> > requested Wangda offline to punt it for 2.7.3.
> >
> > Thanks
> > +Vinod
> >
> > > On Nov 2, 2015, at 12:43 AM, Sunil Govind <sunil.gov...@gmail.com>
> > wrote:
> > >
> > > Thank you.
> > > I will help to backport to 2.7.2.
> > >
> > > Thank you
> > > Sunil
> > >
> > > On Mon, Nov 2, 2015 at 2:10 PM Wangda Tan <wheele...@gmail.com> wrote:
> > >
> > >> +1 to back port it to 2.7.2, marked to 2.7.2-candidate.
> > >>
> > >> On Mon, Nov 2, 2015 at 12:37 AM, Sunil Govind <sunil.gov...@gmail.com
> >
> > >> wrote:
> > >>
> > >>> Thank you Wangda. Sorry to pitch in late here.
> > >>>
> > >>> I feel YARN-3849 is also a good candidate for 2.7.2. This s a bug fix
> > for
> > >>> DRC and preemption.
> > >>>
> > >>> Thank You
> > >>> Sunil
> > >>>
> > >>>
> > >>> On Thu, Oct 29, 2015 at 12:26 PM Naganarasimha G R (Naga) <
> > >>> garlanaganarasi...@huawei.com> wrote:
> > >>>
> > >>>> Thanks for sharing this important viewpoint.
> > >>>>
> > >>>> This sub list of NodeLabels jiras what i have selected is doing
> > minimal
> > >>>> modifications to the core code but tries to increase the usability
> of
> > >>>> NodeLabels and fix some bugs or add missing necessary features
> > >>>> Other additional features which were done for NodeLabels like
> > >>> Distributed
> > >>>> Scheduling or Delegated Centralized are all not included.
> > >>>> May be Wangda could be better judge to further scrutinize the list
> and
> > >>>> select from them or even add to them
> > >>>> My intention here is to only make the Node Labels more usable as
> there
> > >>> has
> > >>>> been long delay for 2.8 and not heard of any approximate dates for
> it.
> > >>>>
> > >>>> Regards,
> > >>>> + Naga
> > >>>>
> > >>>>
> > >>>> ________________________________________
> > >>>> From: Karthik Kambatla [ka...@cloudera.com]
> > >>>> Sent: Thursday, October 29, 2015 04:04
> > >>>> To: yarn-dev@hadoop.apache.org
> > >>>> Cc: Tsuyoshi Ozawa; Vinod Vavilapalli; hdfs-...@hadoop.apache.org;
> > >>>> common-...@hadoop.apache.org; Vinod Kumar Vavilapalli; Wangda Tan
> > >>>> Subject: Re: 2.7.2 release plan
> > >>>>
> > >>>> I would like for us to make sure later maintenance releases are more
> > >>> stable
> > >>>> than previous ones. IMO, increasing stability is more important than
> > >> the
> > >>>> timing of a release.
> > >>>>
> > >>>> Will adding all the patches in 2.7.3 reduce the stability going from
> > >>> 2.7.2
> > >>>> to 2.7.3? If yes, can we just leave them for 2.8.0?
> > >>>>
> > >>>> On Wed, Oct 28, 2015 at 12:07 PM, Naganarasimha G R (Naga) <
> > >>>> garlanaganarasi...@huawei.com> wrote:
> > >>>>
> > >>>>> Yes Vinod & Tsuyoshi,
> > >>>>>
> > >>>>> Within a week merging them would be difficult. I can start
> > >> backporting
> > >>>>> them after 2.7.2 so that it can be ported to 2.7.3 faster, also
> > >> shall i
> > >>>>> apply 2.7.3-candidate labels to them ?
> > >>>>>
> > >>>>> + Naga
> > >>>>> ______________________________
> > >>>>> From: Tsuyoshi Ozawa [oz...@apache.org]
> > >>>>> Sent: Wednesday, October 28, 2015 23:13
> > >>>>> To: Vinod Vavilapalli
> > >>>>> Cc: yarn-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org;
> > >>>>> common-...@hadoop.apache.org; Vinod Kumar Vavilapalli; Wangda Tan;
> > >>>>> Tsuyoshi Ozawa; Naganarasimha G R (Naga)
> > >>>>> Subject: Re: 2.7.2 release plan
> > >>>>>
> > >>>>> Vinod,
> > >>>>>
> > >>>>> Thank you for taking care of this. I've checked the list of
> changes.
> > >>>>> As a result, I agree that we don't have enough time to backport
> these
> > >>>>> changes into 2.7.2 by this weekend. For a fast move, it's better
> > >>>>> suggestion to me to backport these tickets into 2.7.3.
> > >>>>>
> > >>>>> Best,
> > >>>>> - Tsuyoshi
> > >>>>>
> > >>>>> On Thu, Oct 29, 2015 at 2:19 AM, Vinod Vavilapalli
> > >>>>> <vino...@hortonworks.com> wrote:
> > >>>>>> Tsuyoshi / Wangda / Naga,
> > >>>>>>
> > >>>>>> This looks too big of a list to me if we have to cut an RC by this
> > >>>>> weekend per my plan.
> > >>>>>>
> > >>>>>> I’d suggest a fast move on things you think are low risk enough
> and
> > >>>> punt
> > >>>>> everything else for next release.
> > >>>>>>
> > >>>>>> Thanks
> > >>>>>> +Vinod
> > >>>>>>
> > >>>>>>> On Oct 28, 2015, at 3:08 AM, Naganarasimha G R (Naga) <
> > >>>>> garlanaganarasi...@huawei.com> wrote:
> > >>>>>>>
> > >>>>>>> Thanks Tsuyoshi,
> > >>>>>>> If required even i can pitch in :)
> > >>>>>>> Additional to this we added the support in Mapreduce for labels
> in
> > >>>>> MAPREDUCE-6304,
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> + Naga
> > >>>>>>> ________________________________________
> > >>>>>>> From: Tsuyoshi Ozawa [oz...@apache.org]
> > >>>>>>> Sent: Wednesday, October 28, 2015 14:28
> > >>>>>>> To: yarn-dev@hadoop.apache.org
> > >>>>>>> Cc: hdfs-...@hadoop.apache.org; common-...@hadoop.apache.org;
> > >> Vinod
> > >>>>> Kumar Vavilapalli; Wangda tan
> > >>>>>>> Subject: Re: 2.7.2 release plan
> > >>>>>>>
> > >>>>>>> Thank you for reporting, Naganarasimha.
> > >>>>>>> Vinod and Wangda, I will help you to backport these changes.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> - Tsuyoshi
> > >>>>>>>
> > >>>>>>> On Wed, Oct 28, 2015 at 2:57 PM, Naganarasimha G R (Naga)
> > >>>>>>> <garlanaganarasi...@huawei.com> wrote:
> > >>>>>>>> Hi Vinod, & Wangda
> > >>>>>>>>
> > >>>>>>>> I think it would be good to backport, following jira's related
> to
> > >>>>> NodeLabels as it will improve debug ability and usability of
> > >> NodeLabels
> > >>>>>>>> --------------------------------
> > >>>>>>>> Key Summary
> > >>>>>>>> --------------------------------
> > >>>>>>>> YARN-4215 YARN-2492 RMNodeLabels Manager Need to verify
> and
> > >>>>> replace node labels for the only modified Node Label Mappings in
> the
> > >>>> request
> > >>>>>>>> YARN-4162 YARN-2492 CapacityScheduler: Add resource usage
> > >> by
> > >>>>> partition and queue capacity by partition to REST API
> > >>>>>>>> YARN-4140 YARN-2492 RM container allocation delayed incase
> > >> of
> > >>>>> app submitted to Nodelabel partition
> > >>>>>>>> YARN-3717 YARN-2492 Expose app/am/queue's
> > >>> node-label-expression
> > >>>>> to RM web UI / CLI / REST-API
> > >>>>>>>> YARN-3647 YARN-2492 RMWebServices api's should use updated
> > >>> api
> > >>>>> from CommonNodeLabelsManager to get NodeLabel object
> > >>>>>>>> YARN-3593 YARN-2492 Add label-type and Improve
> > >>>>> "DEFAULT_PARTITION" in Node Labels Page
> > >>>>>>>> YARN-3583 YARN-2492 Support of NodeLabel object instead of
> > >>>> plain
> > >>>>> String in YarnClient side.
> > >>>>>>>> YARN-3581 YARN-2492 Deprecate
> -directlyAccessNodeLabelStore
> > >>> in
> > >>>>> RMAdminCLI
> > >>>>>>>> YARN-3579 YARN-2492 CommonNodeLabelsManager should support
> > >>>>> NodeLabel instead of string label name when getting
> > >>>>> node-to-label/label-to-label mappings
> > >>>>>>>> YARN-3565 YARN-2492
> > >>>>> NodeHeartbeatRequest/RegisterNodeManagerRequest should use
> NodeLabel
> > >>>> object
> > >>>>> instead of String
> > >>>>>>>> YARN-3521 YARN-2492 Support return structured NodeLabel
> > >>> objects
> > >>>>> in REST API
> > >>>>>>>> YARN-3362 YARN-2492 Add node label usage in RM
> > >>>> CapacityScheduler
> > >>>>> web UI
> > >>>>>>>> YARN-3326 YARN-2492 Support RESTful API for
> > >> getLabelsToNodes
> > >>>>>>>> YARN-3216 YARN-2492 Max-AM-Resource-Percentage should
> > >> respect
> > >>>>> node labels
> > >>>>>>>> YARN-3136 YARN-3091 getTransferredContainers can be a
> > >>>> bottleneck
> > >>>>> during AM registration
> > >>>>>>>>
> > >>>>>>>> Please inform if any support is required to backport them to
> > >> 2.7.2
> > >>>>>>>>
> > >>>>>>>> Regards,
> > >>>>>>>> + Naga
> > >>>>>>>> ________________________________________
> > >>>>>>>> From: Kihwal Lee [kih...@yahoo-inc.com.INVALID]
> > >>>>>>>> Sent: Tuesday, October 27, 2015 20:42
> > >>>>>>>> To: hdfs-...@hadoop.apache.org; common-...@hadoop.apache.org
> > >>>>>>>> Cc: Chris Nauroth; yarn-dev@hadoop.apache.org;
> > >>>>> mapreduce-...@hadoop.apache.org; Vinod Kumar Vavilapalli; Ming Ma
> > >>>>>>>> Subject: Re: 2.7.2 release plan
> > >>>>>>>>
> > >>>>>>>> I think we need HDFS-8950 and HDFS-7725 in 2.7.2.It should be
> > >> easy
> > >>>> to
> > >>>>> backport/cherry-pick HDFS-7725. For HDFS-8950, it will be nice if
> > >> Ming
> > >>>> can
> > >>>>> chime in.
> > >>>>>>>> Kihwal
> > >>>>>>>>
> > >>>>>>>> From: Tsuyoshi Ozawa <oz...@apache.org>
> > >>>>>>>> To: "common-...@hadoop.apache.org" <
> common-...@hadoop.apache.org
> > >>>
> > >>>>>>>> Cc: Chris Nauroth <cnaur...@hortonworks.com>; "
> > >>>>> yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>; "
> > >>>>> hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>; "
> > >>>>> mapreduce-...@hadoop.apache.org" <mapreduce-...@hadoop.apache.org
> >;
> > >>>> Vinod
> > >>>>> Kumar Vavilapalli <vino...@apache.org>
> > >>>>>>>> Sent: Tuesday, October 27, 2015 2:39 AM
> > >>>>>>>> Subject: Re: 2.7.2 release plan
> > >>>>>>>>
> > >>>>>>>> Vinod and Chris,
> > >>>>>>>>
> > >>>>>>>> Thanks for your reply. I'll do also backport not only bug fixes
> > >> but
> > >>>>>>>> also documentations(I think 2.7.2 includes them). It helps users
> > >> a
> > >>>> lot.
> > >>>>>>>>
> > >>>>>>>> Best,
> > >>>>>>>> - Tsuyoshi
> > >>>>>>>>
> > >>>>>>>> On Tuesday, 27 October 2015, Vinod Vavilapalli <
> > >>>>> vino...@hortonworks.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> +1.
> > >>>>>>>>>
> > >>>>>>>>> Thanks
> > >>>>>>>>> +Vinod
> > >>>>>>>>>
> > >>>>>>>>>> On Jul 16, 2015, at 8:18 AM, Chris Nauroth <
> > >>>> cnaur...@hortonworks.com
> > >>>>>>>>> <javascript:;>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> I'd be comfortable with inclusion of any doc-only patch in
> > >> minor
> > >>>>>>>>> releases.
> > >>>>>>>>>> There is a lot of value to end users in pushing documentation
> > >>> fixes
> > >>>>> as
> > >>>>>>>>>> quickly as possible, and they don't bear the same risk of
> > >>>>> regressions or
> > >>>>>>>>>> incompatibilities as code changes.
> > >>>>>>>>>>
> > >>>>>>>>>> --Chris Nauroth
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 7/16/15, 12:38 AM, "Tsuyoshi Ozawa" <oz...@apache.org
> > >>>>> <javascript:;>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi,
> > >>>>>>>>>>>
> > >>>>>>>>>>> thank you for starting the discussion about 2.7.2 release.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> The focus obviously is to have blocker issues [2], bug-fixes
> > >>> and
> > >>>>> *no*
> > >>>>>>>>>>> features / improvements.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I've committed YARN-3170, which is an improvement of
> > >>>> documentation.
> > >>>>> I
> > >>>>>>>>>>> thought documentation pages which can be fit into branch-2.7
> > >> can
> > >>>> be
> > >>>>>>>>>>> included easily. Should I revert it?
> > >>>>>>>>>>>
> > >>>>>>>>>>>>> I need help from all committers in automatically
> > >>>>>>>>>>> merging in any patch that fits the above criterion into 2.7.2
> > >>>>> instead of
> > >>>>>>>>>>> only on trunk or 2.8.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Sure, I'll try my best.
> > >>>>>>>>>>>
> > >>>>>>>>>>>> That way we can include not only blocker but also critical
> > >> bug
> > >>>>> fixes to
> > >>>>>>>>>>>> 2.7.2 release.
> > >>>>>>>>>>>
> > >>>>>>>>>>> As Vinod mentioned, we should also apply major bug fixes into
> > >>>>>>>>> branch-2.7.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> - Tsuyoshi
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Jul 16, 2015 at 3:52 PM, Akira AJISAKA
> > >>>>>>>>>>> <ajisa...@oss.nttdata.co.jp <javascript:;>> wrote:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> Thanks Vinod for starting 2.7.2 release plan.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> The focus obviously is to have blocker issues [2],
> bug-fixes
> > >>> and
> > >>>>> *no*
> > >>>>>>>>>>>>> features / improvements.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Can we adopt the plan as Karthik mentioned in "Additional
> > >>>>> maintenance
> > >>>>>>>>>>>> releases for Hadoop 2.y versions" thread? That way we can
> > >>> include
> > >>>>> not
> > >>>>>>>>>>>> only
> > >>>>>>>>>>>> blocker but also critical bug fixes to 2.7.2 release.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> In addition, branch-2.7 is a special case. (2.7.1 is the
> > >> first
> > >>>>> stable
> > >>>>>>>>>>>> release) Therefore I'm thinking we can include major bug
> > >> fixes
> > >>> as
> > >>>>> well.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regards,
> > >>>>>>>>>>>> Akira
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 7/16/15 04:13, Vinod Kumar Vavilapalli wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks everyone for the push on 2.7.1! Branch-2.7 is now
> > >> open
> > >>>> for
> > >>>>>>>>>>>>> commits
> > >>>>>>>>>>>>> to a 2.7.2 release. JIRA also now has a 2.7.2 version for
> > >> all
> > >>>> the
> > >>>>>>>>>>>>> sub-projects.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Continuing the previous 2.7.1 thread on steady maintenance
> > >>>>> releases
> > >>>>>>>>>>>>> [1],
> > >>>>>>>>>>>>> we
> > >>>>>>>>>>>>> should follow up 2.7.1 with a 2.7.2 within 4 weeks. Earlier
> > >> I
> > >>>>> tried a
> > >>>>>>>>>>>>> 2-3
> > >>>>>>>>>>>>> week cycle for 2.7.1, but it seems to be impractical given
> > >> the
> > >>>>>>>>>>>>> community
> > >>>>>>>>>>>>> size. So, I propose we target a release by the end for 4
> > >> weeks
> > >>>>> from
> > >>>>>>>>>>>>> now,
> > >>>>>>>>>>>>> starting the release close-down within 2-3 weeks.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> The focus obviously is to have blocker issues [2],
> bug-fixes
> > >>> and
> > >>>>> *no*
> > >>>>>>>>>>>>> features / improvements. I need help from all committers in
> > >>>>>>>>>>>>> automatically
> > >>>>>>>>>>>>> merging in any patch that fits the above criterion into
> > >> 2.7.2
> > >>>>> instead
> > >>>>>>>>>>>>> of
> > >>>>>>>>>>>>> only on trunk or 2.8.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thoughts?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> +Vinod
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> [1] A 2.7.1 release to follow up 2.7.0
> > >>>>>>>>>>>>> http://markmail.org/message/zwzze6cqqgwq4rmw
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> [2] 2.7.2 release blockers:
> > >>>>>>>>>>>>> https://issues.apache.org/jira/issues/?filter=12332867
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> >
> >
>
