Hi Frank Thanks for sharing more details. Let me try this combination and I might be wrong, so pls correct me. And I think sharable node-label could help here.
Labels: Node1-4 = default label Node8-9 = "special" label Queues: "ProdQ" accessable-labels is "" (only default label) "TestQ" accessable-labels is "" (only default label) "LabeledQ" accessable-labels "special" Capacity Per Queue: "ProdQ" capacity=50% max-capacity=100% "TestQ" capacity=50% max-capacity=50% "LabeledQ" special.capacity=100% special.max-capacity=100% Various Choices: * Jobs in ProdQ is assured with 50% of default label resources and it can go to 100% if there are no resource running in TestQ * Jobs in TestQ can only get 50% of default label resources. * If jobs in ProdQ or TestQ needed to make use of "special" label machines, it is only possible when there are resource available in "special" label. "special" is a non-exclusive label which can share its resource with "default" label. * Any job submitted in "LabeledQ" is assured with 100% of special resources and can use 100% if nothing is there. I think preemption could be made optional here. If Inter queue preemption is enabled, we can enforce a normalization faster for default label, else apps might need to wait. We could also try another approach as I shared in an earlier mail. But it ensure some % of resources for ProdQ and TestQ in LabeledQ which may not be suitable. Thanks Sunil On Tue, Oct 11, 2016 at 10:38 PM Frank Luo <j...@merkleinc.com> wrote: > > > > > > > > > Hah, how so? I am confused as I was under impression that I needed sharing > but not preemption. > > > > Let’s model this out. > > > > Assuming I got 4 “normal” machines node1-4, and two special node8 and > node9 where JobA can be executed on. > > > > And I need two queues, ProdQ and TestQ equally sharing Node1-4, and a > “LabeledQ” with node8/9. > > > > When ProdQ is full, it can overflow to TestQ and further to LabeledQ. If > TestQ is full, the tasks stay in TestQ, or optionally overflow to LabeledQ > (either way > is fine as long as it doesn’t go to ProdQ). And when JobA is running, it > can only go to LabelledQ. If something else is on LabelledQ, JobA waits. > > > > Do you mind to illustrate how to config the queues to achieve what I am > looking for? > > > > Thank you Sunil. > > > > *From:* Sunil Govind [mailto:sunil.gov...@gmail.com] > > > *Sent:* Tuesday, October 11, 2016 11:44 AM > > > > *To:* Frank Luo <j...@merkleinc.com>; user@hadoop.apache.org > > *Subject:* Re: how to add a shareable node label? > > > > > Hi Frank > > > > > > > Extremely sorry for the delay.. > > > > > > > > Yes, you are correct. Sharing feature of node label is not needed in your > case. > > > > Existing node labels and a queue model could solve the problem. > > > > > > > > Thanks > > > > Sunil > > > > > > > > On Fri, Oct 7, 2016 at 11:59 PM Frank Luo <j...@merkleinc.com> wrote: > > > > > > That is correct, Sunil. > > > > Just to confirm, the Node Labeling feature on 2.8 or 3.0 alpha won’t > satisfy > my need, right? > > > > *From:* > Sunil Govind [mailto:sunil.gov...@gmail.com] > > > *Sent:* Friday, October 07, 2016 12:09 PM > > > > > > > > *To:* Frank Luo <j...@merkleinc.com>; > user@hadoop.apache.org > > *Subject:* Re: how to add a shareable node label? > > > > > > > > > HI Frank > > > > > > > In that case, preemption may not be needed. So over-utilizing resources of > queueB will be running till it completes. Since queueA is under served, > then any next free container could > go to queueA which is for Job_A. > > > > > > > > Thanks > > > > Sunil > > > > > > > > On Fri, Oct 7, 2016 at 9:58 PM Frank Luo <j...@merkleinc.com> wrote: > > > > > > Sunil, > > > > Your description pretty much matches my understanding. Except for “Job_A > will have to run as per its schedule w/o any delay”. My situation is that > Job_A can be delayed. As long as it runs in queueA, I am happy. > > > > Just as you said, processes normally running in queueB might not be > preemptable. > So if they overflow to queueA then got preempted, then that is not good. > > > > *From:* > Sunil Govind [mailto:sunil.gov...@gmail.com] > > > *Sent:* Friday, October 07, 2016 10:50 AM > > > > > > > > *To:* Frank Luo <j...@merkleinc.com>; > user@hadoop.apache.org > > > > > > *Subject:* > Re: how to add a shareable node label? > > > > > > > > > > HI Frank > > > > > > > Thanks for the details. > > > > > > > > I am not quite sure if I understood you problem correctly. I think you are > looking for a solution to ensure that Job_A will have to run as per its > schedule w/o any delay. Meantime > you also do not want to waste resources on those high end machine where > Job_A is running. > > > > > > > > I think you still need node label exclusivity here since there is h/w > dependency. But if you have 2 queues' which are shared to use "labelA" > here, then always "Job_A" can be planned > to run in that queue, say "queueA". Other jobs could be run in "queueB" > here. So if you tune capacities and if preemption is enabled per queue > level, overutilized resources used by "queueB" could be preempted for > "Job_A". > > > > > > > > But if your sharable jobs are like some linux jobs which should not be > preempted, then this may be only a half solution. > > > > > > > > Thanks > > > > > > Sunil > > > > > > > > On Fri, Oct 7, 2016 at 7:36 AM Frank Luo <j...@merkleinc.com> wrote: > > > > > > Sunil, > > > > You confirmed my understanding. I got the understanding by reading the docs > and haven’t really tried 2.8 or 3.0-alphal. > > > > My situation is that I am in a multi-tenant env, and got several very > powerful > machines with expensive licenses to run a particular linux job, let’s say > Job_A. But the job is executed infrequently, so I want to let other jobs to > use the machines when Job_A is not running. In the meaning time, I am not > powerful enough to force all other > jobs to be preemptable. As matter of fact, I know they have Hadoop jobs > inserting into sql-server, or just pure linux jobs that are not preemptable > in nature. So preempt jobs is not an option for me. > > > > I hope it makes sense. > > > > Frank > > > > *From:* > Sunil Govind [mailto:sunil.gov...@gmail.com] > > > *Sent:* Thursday, October 06, 2016 2:15 PM > > > > > > > > *To:* Frank Luo <j...@merkleinc.com>; > user@hadoop.apache.org > > *Subject:* Re: how to add a shareable node label? > > > > > > > > > HI Frank > > > > > > > Ideally those containers will be preempted if there are unsatisfied demand > for "configured label". > > > > > > > > I could explain this: > > > > "labelA" has few empty resources. All nodes under "default" label is > used. Hence a new application which is submitted to "default" label has to > wait. But if "labelA" is non-exclusive > and there are some free resources, this new application can run on > "labelA". > > > > However if there are some more new apps submitted to "labelA", and if > there are no more resources available in "labelA", then it may preempt > containers from the app which was sharing > containers earlier. > > > > > > > > May be you could share some more information so tht it may become more > clear. Also I suppose you are running this in hadoop 3 alpha1 release. > please correct me if I m wrong. > > > > > > > > Thanks > > > > Sunil > > > > > > > > On Thu, Oct 6, 2016 at 9:44 PM Frank Luo <j...@merkleinc.com> wrote: > > > > > > Thanks Sunil. > > > > > Ø > 3. If there is any future ask for those resources , we will preempt the > non labeled apps and give them back to labeled apps. > > > > Unfortunately, I am still not able to use it, because of the preemptive > behavior. > The jobs that steals labelled resources are not preemptable, and I’d > rather waiting instead of killing. > > > > *From:* > Sunil Govind [mailto:sunil.gov...@gmail.com] > > > *Sent:* Thursday, October 06, 2016 1:59 AM > > > > > > > > *To:* Frank Luo <j...@merkleinc.com>; > user@hadoop.apache.org > > > > > > *Subject:* > Re: how to add a shareable node label? > > > > > > > > > Hi Frank > > > I think as of today this is not possible. You could try and experience the > "non-exlusive" feature of node-label which will officially come in 2.8 > soon. Or you can try it in "Hadoop > 3 alpha1" release too if its fine to check. YARN-3214 > <https://issues.apache.org/jira/browse/YARN-3214> has > the details for the nodelabel sharing concept. > > > > > > > > Thanks > > > > Sunil > > > > > > > > On Wed, Oct 5, 2016 at 8:14 PM Frank Luo <j...@merkleinc.com> wrote: > > > > > > Sunil, thanks for responding. > > > > So is there any way to dedicate one kind of jobs to certain machines, then > having > those machines be shared if no dedicated job running? > > > > *From:* > Sunil Govind [mailto:sunil.gov...@gmail.com] > > > *Sent:* Wednesday, October 05, 2016 12:50 AM > > *To:* Frank Luo <j...@merkleinc.com>; > user@hadoop.apache.org; > u...@yarn.apache.org > > > > > > > > *Subject:* Re: how to add a shareable node label? > > > > > > > > > Hi Frank, > > > > > > > > > > > > > As far as I checked, all labels are "exclusive" in 2.7. In upcoming 2.8 > release, we can get "non-exclusive" or sharable node labels. > > > > > > > > Thanks > > > > Sunil > > > > > > > > > > > > On Wed, Oct 5, 2016 at 8:40 AM Frank Luo <j...@merkleinc.com> wrote: > > > > > > I am using Hadoop 2.7.3, when I run: > > > $ yarn rmadmin -addToClusterNodeLabels "Label1(exclusive=false)" > > > > I got an error as: > > > … addToClusterNodeLabels: java.io.IOException: label name should only > contains {0-9, a-z, A-Z, -, _} and should not started with {-,_} > > > > If I just use “Label1”, it will work fine, but I want a shareable one. > > > > > Anyone knows a better way to do it? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Access the Q2 2016 Digital Marketing Report for a fresh set of trends and > benchmarks in digital marketing* > <http://www2.merkleinc.com/l/47252/2016-07-26/47gt7c> > > > *Download our latest report titled “The Case for Change: Exploring the > Myths of Customer-Centric Transformation”* > <http://www2.merkleinc.com/l/47252/2016-08-04/4b9p7c> > > This email and any attachments transmitted with it are intended for use by > the intended recipient(s) only. If you have received this email in error, > please notify the sender > immediately and then delete it. If you are not the intended recipient, you > must not keep, use, disclose, copy or distribute this email without the > author’s prior permission. We take precautions to minimize the risk of > transmitting software viruses, but we > advise you to perform your own virus checks on any attachment to this > message. We cannot accept liability for any loss or damage caused by > software viruses. The information contained in this communication may be > confidential and may be subject to the attorney-client > privilege. > >
