Thanks for the input Chinmay. The last parameter true/false is for relaxing the constraint. If true, the affinity rule would be relaxed if constraint cannot be satisfied. And if false, application will keep requesting and waiting to launch containers till all rules are satisfied.
Regarding the dag.setAffinity APIs. I can add those couple APIs in LogicalPlan as a wrapper that internally can set the Dag Attributes. Probably yes, that would be easier to use, to set affinity directly on dag object. Then there would 2 ways to set affinity rules: 1. Through DAG context attribute (Either through code or through properties.xml) 2. By calling setAffinity on dag. Thanks, Isha On Wed, Feb 24, 2016 at 10:11 PM, Chinmay Kolhatkar <[email protected]> wrote: > I agree with Pramod. We should probably have single interface for > specifying both if at all possible. This is in interest of not confusing > users with two different configurations at the same time make a > locality/affinity API to apear similar to what is out there in the word at > the moment. > > Though it might be early to deprecate stream locality, but after some time > based on user's experience we should consider deprecating Stream Locality > API. > > @Isha, nice APIs. Questions/suggestions regarding this: > 1. What does the "false" signify for AffinityRule constructor. > 2. This is a suggestion. Instead of having a List<AffinityRule> and then > setting this object to attribute of the DAG, should we have an API in > LogicalPlan like following: > > dag.setAffinity(<operator1>, <operator2>, <locality_type>, false/true) > > dag.setAntiAffinity(<operator1>, <operator2>, false/true) // Not sure > what false/true mean, added for completion. But the point here is to remove > locality for anti-affinity. > > > Thanks, > Chinmay. > > > > On Thu, Feb 25, 2016 at 5:17 AM, Pramod Immaneni <[email protected]> > wrote: > > > You could specify it by specifying the names of all operators that are > part > > of the stream. > > > > On Wed, Feb 24, 2016 at 3:44 PM, Amol Kekre <[email protected]> > wrote: > > > > > These are two diff ways of specifying something similar, but not always > > > same. For example, how do I specify the following > > > > > > This stream has way to high throughput. Force all operators that > connect > > to > > > it today (or in future) to be container local. > > > > > > With stream locality I get that as operator ports are added to the > > stream, > > > they automatically inherit stream locality. > > > > > > Thks, > > > Amol > > > > > > On Wed, Feb 24, 2016 at 3:37 PM, Pramod Immaneni < > [email protected] > > > > > > wrote: > > > > > > > To explain the affinity support below seems to be a super-set that > can > > > > consume the stream locality setting whether that was the original > > > intention > > > > or not. > > > > > > > > On Wed, Feb 24, 2016 at 3:35 PM, Pramod Immaneni < > > [email protected] > > > > > > > > wrote: > > > > > > > > > Why would we want two different ways of specifying the same thing? > > > > > > > > > > On Wed, Feb 24, 2016 at 3:29 PM, Amol Kekre <[email protected]> > > > > wrote: > > > > > > > > > >> Stream locality is a stream attribute. Affinity is operator > > attribute. > > > > >> Technically they may not be connected. Though operator affinity > can > > be > > > > >> extended to specify a situation that implicity covers stream > > locality, > > > > it > > > > >> may be better to let user say the two following statements > > > > independently. > > > > >> > > > > >> - This stream is heavy on I/O and I want it to be ..._local // > This > > > > >> statement is independent of future additions of operators to this > > > stream > > > > >> - I need these logical operators to not be (or be) on the same > node > > > > >> > > > > >> Do note that container_local or node_local stream localities only > > > relate > > > > >> to > > > > >> connected operators. > > > > >> > > > > >> Thks > > > > >> Amol > > > > >> > > > > >> > > > > >> On Wed, Feb 24, 2016 at 1:54 PM, Pramod Immaneni < > > > > [email protected]> > > > > >> wrote: > > > > >> > > > > >> > Would it make sense to deprecate stream locality in favor of > this? > > > > >> > > > > > >> > On Tue, Feb 23, 2016 at 6:05 PM, Isha Arkatkar < > > > [email protected]> > > > > >> > wrote: > > > > >> > > > > > >> > > Hi Pramod, > > > > >> > > > > > > >> > > We can have a list or operators or regex to specify > operators > > > > list > > > > >> for > > > > >> > > affinity rule. Though, internal implementation will translate > > both > > > > >> these > > > > >> > > into list of pairs. Since, having pair of operators helps in > > > > >> validation > > > > >> > > phase. > > > > >> > > > > > > >> > > And yes, validation phase will catch conflicting locality or > > > > affinity > > > > >> > > rules and throw validation exception. > > > > >> > > > > > > >> > > Thanks for input regarding unifiers. It is probably alright > to > > > not > > > > >> > include > > > > >> > > it in the affinity settings. Will leave the current > > implementation > > > > as > > > > >> is. > > > > >> > > > > > > >> > > Thanks! > > > > >> > > Isha > > > > >> > > > > > > >> > > > > > > >> > > On Tue, Feb 23, 2016 at 5:56 PM, Pramod Immaneni < > > > > >> [email protected] > > > > >> > > > > > > >> > > wrote: > > > > >> > > > > > > >> > > > Isha, > > > > >> > > > > > > > >> > > > If the implementation would support (eventually) affinity > > > between > > > > >> more > > > > >> > > than > > > > >> > > > 2 logical operators, instead of relying on specifying it as > > > > multiple > > > > >> > > rules > > > > >> > > > why not make provision in the API to specify more than 2 > > logical > > > > >> > > operators? > > > > >> > > > If there is a stream locality rule that conflicts with the > > > > affinity > > > > >> > rule > > > > >> > > > will the validation catch this? With unifiers the common > case > > of > > > > MxN > > > > >> > > > locality is already chosen based on downstream operator. In > > > other > > > > >> cases > > > > >> > > > such as single downstream or cascade case I am not sure it > is > > > all > > > > >> the > > > > >> > > > important to have it. > > > > >> > > > > > > > >> > > > Thanks > > > > >> > > > > > > > >> > > > On Tue, Feb 23, 2016 at 5:37 PM, Isha Arkatkar < > > > > >> [email protected]> > > > > >> > > > wrote: > > > > >> > > > > > > > >> > > > > Hi all, > > > > >> > > > > > > > > >> > > > > I have opened a review only pull request for handling > > > > affinity > > > > >> > > rules > > > > >> > > > in > > > > >> > > > > Apex: > > > > >> > > > > https://github.com/apache/incubator-apex-core/pull/234 > > > > >> > > > > > > > > >> > > > > Just wanted to confirm with everyone, if the APIs for > > > > specifying > > > > >> > > > affinity > > > > >> > > > > rules look alright. Also, wanted to get views on how > > unifiers > > > > >> should > > > > >> > be > > > > >> > > > > handled in case of operator affinity rules. More details > > > follow: > > > > >> > > > > > > > > >> > > > > I have added a list of affinity rules as an attribute in > > Dag > > > > >> > context. > > > > >> > > > For > > > > >> > > > > now, have added AffinityRule to be specified for operator > > > pairs. > > > > >> We > > > > >> > can > > > > >> > > > add > > > > >> > > > > regex support in next iteration. > > > > >> > > > > > > > > >> > > > > Here is sample usage: > > > > >> > > > > > > > > >> > > > > List<AffinityRule> rules = new ArrayList<>(); > > > > >> > > > > // To add node locality between 2 not connected > > operators > > > > rand > > > > >> > and > > > > >> > > > > console: > > > > >> > > > > rules.add(new AffinityRule(Type.AFFINITY, new > > OperatorPair( > > > > >> > "rand", > > > > >> > > > > "console"), Locality.NODE_LOCAL, false)); > > > > >> > > > > > > > > >> > > > > // To add anti-affinity between all partitions of rand > > > > operator > > > > >> > with > > > > >> > > > all > > > > >> > > > > partitions of passThru operator: > > > > >> > > > > rules.add(new AffinityRule(Type.ANTI_AFFINITY, new > > > > >> OperatorPair( > > > > >> > > > "rand", > > > > >> > > > > "passThru"), Locality.NODE_LOCAL, false)); > > > > >> > > > > > > > > >> > > > > // To add anti-affinity between partitions of passThru > > > > >> operator, > > > > >> > > give > > > > >> > > > > same operator name in pair: > > > > >> > > > > rules.add(new AffinityRule(Type.ANTI_AFFINITY, new > > > > >> OperatorPair( > > > > >> > > > > "passThru", "passThru"), Locality.NODE_LOCAL, false)); > > > > >> > > > > > > > > >> > > > > // Set the rules in dag context > > > > >> > > > > dag.setAttribute(DAGContext.AFFINITY_RULES, rules); > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > Please find a sample application using these affinity > rules > > > > here: > > > > >> > > > > > > > > >> > https://github.com/ishark/Apex-Samples/tree/master/affinity-example > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > The actual implementation of affinity rules heavily > depends > > > node > > > > >> > > specific > > > > >> > > > > requests, that was already being handled in > > > > >> > StreamingAppMasterService. > > > > >> > > To > > > > >> > > > > handle node requests for cloudera, I have added an > override > > to > > > > >> > > blacklist > > > > >> > > > > all other nodes except the ones where container request is > > to > > > be > > > > >> > > issued. > > > > >> > > > > > > > > >> > > > > There is one open question I wanted to bring up: In case > of > > > > >> > partitioned > > > > >> > > > > operators, should the unifiers also follow affinity or > > > > >> anti-affinity > > > > >> > > > rules? > > > > >> > > > > Or should they be independent. For now, I have kept them > > > > >> independent > > > > >> > > and > > > > >> > > > > only the actual operators follow affinity rules. Please > > > suggest > > > > >> what > > > > >> > > > would > > > > >> > > > > make more sense from user's specification perspective. > > > > >> > > > > > > > > >> > > > > Thanks, > > > > >> > > > > Isha > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > On Mon, Feb 1, 2016 at 2:12 PM, Isha Arkatkar < > > > > >> [email protected]> > > > > >> > > > > wrote: > > > > >> > > > > > > > > >> > > > > > Hi folks, > > > > >> > > > > > > > > > >> > > > > > Summarizing the proposal for affinity/anti-affinity > > rules > > > > in > > > > >> > apex > > > > >> > > as > > > > >> > > > > > per discussions in this mail thread. Please suggest if I > > > > missed > > > > >> > > > > something. > > > > >> > > > > > > > > > >> > > > > > * For configuration:* > > > > >> > > > > > > > > > >> > > > > > - We will have application level > affinity/anti-affinity > > > > rules. > > > > >> > For > > > > >> > > > the > > > > >> > > > > > first iteration, we can support for (anti-)affinity > among > > > > >> operators > > > > >> > > > > within > > > > >> > > > > > single application. We can revisit it in the next > > iteration > > > to > > > > >> > > support > > > > >> > > > > > across applications. > > > > >> > > > > > > > > > >> > > > > > - Each rule will consist of 4 components: > > > > >> > > > > > <List of operators>, AFFINITY/ANTI-AFFINITY, > > > > >> STRICT/RELAXED > > > > >> > > > > policy, > > > > >> > > > > > CONTAINER/NODE/RACK > > > > >> > > > > > I have checked that Apex supports container locality > > > between 2 > > > > >> > > > operators > > > > >> > > > > > that are not connected by Steam. Though did not find an > > API > > > > for > > > > >> > > setting > > > > >> > > > > it > > > > >> > > > > > through application. So, affinity rules can cover > > container > > > > >> > affinity > > > > >> > > as > > > > >> > > > > > well. > > > > >> > > > > > We will not support THREAD locality in app level > affinity > > or > > > > >> > > > > > anti-affinity, since it does not apply to unconnected > > > > operators. > > > > >> > > > > > > > > > >> > > > > > The list of operators can also be specified by regex > > > > >> > > > > > > > > > >> > > > > > *For Implementation: * > > > > >> > > > > > > > > > >> > > > > > - I have verified that Node specific requests are > > honored > > > by > > > > >> > > > > HortonWorks > > > > >> > > > > > for both Fair and Capacity scheduler. So we can go with > > the > > > > >> > approach > > > > >> > > > of > > > > >> > > > > > node specific container requests to implement affinity > > > rules. > > > > >> > > > > > > > > > >> > > > > > - However, we may need to fall back on blacklist > approach > > > for > > > > >> CDH > > > > >> > > > > distro, > > > > >> > > > > > as node specific requests may not work well. As: > > > > >> > > > > > -> With Fair scheduler, node specific requests > > are > > > > not > > > > >> > > honored > > > > >> > > > > in > > > > >> > > > > > CDH 4. Application master keeps sending requests to > Yarn, > > > but > > > > no > > > > >> > > > > containers > > > > >> > > > > > are allocated. > > > > >> > > > > > -> I could not yet verify the same with Fair > and > > > > >> Capacity > > > > >> > > > > > Scheduler on CDH 5. Ran into some configuration issues > > when > > > > >> setting > > > > >> > > up > > > > >> > > > > CDH > > > > >> > > > > > 5.5.1. Will update the thread with findings. > > > > >> > > > > > > > > > >> > > > > > I will try to build a prototype for affinity rules > with > > > > these > > > > >> > > things > > > > >> > > > in > > > > >> > > > > > mind. Please let me know if anyone would like to > > collaborate > > > > on > > > > >> > this > > > > >> > > > > > effort! :) > > > > >> > > > > > > > > > >> > > > > > Thanks! > > > > >> > > > > > Isha > > > > >> > > > > > > > > > >> > > > > > On Mon, Jan 25, 2016 at 10:23 AM, Vlad Rozov < > > > > >> > > [email protected]> > > > > >> > > > > > wrote: > > > > >> > > > > > > > > > >> > > > > >> Yes, with ability to refer not only to a host name, but > > to > > > > >> > operators > > > > >> > > > > >> (possibly both in the same and other applications). In > > > > general > > > > >> > this > > > > >> > > > > should > > > > >> > > > > >> allow to specify both affinity and anti affinity using > > > > >> > expressions. > > > > >> > > > For > > > > >> > > > > >> example LOCALITY_CONTAINER = "{app1.op1 or app1.op2} > and > > > > >> > > {!app2.op1}". > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> On 1/25/16 09:39, Thomas Weise wrote: > > > > >> > > > > >> > > > > >> > > > > >>> Agreed. Originally I thought you wanted both operators > > on > > > > the > > > > >> > same > > > > >> > > > > host. > > > > >> > > > > >>> To > > > > >> > > > > >>> pin them to the same container the same idea can be > > > applied, > > > > >> > > through > > > > >> > > > a > > > > >> > > > > >>> "LOCALITY_CONTAINER" attribute. > > > > >> > > > > >>> > > > > >> > > > > >>> > > > > >> > > > > >>> On Mon, Jan 25, 2016 at 9:18 AM, Vlad Rozov < > > > > >> > > [email protected] > > > > >> > > > > > > > > >> > > > > >>> wrote: > > > > >> > > > > >>> > > > > >> > > > > >>> Works as a workaround and requires two extra ports > that > > > > >> operators > > > > >> > > > > >>>> designer > > > > >> > > > > >>>> may not necessarily provide out of the box. We may > > apply > > > > the > > > > >> > same > > > > >> > > > hack > > > > >> > > > > >>>> to > > > > >> > > > > >>>> anti-affinity and request that any two operators with > > > > >> > > anti-affinity > > > > >> > > > > rule > > > > >> > > > > >>>> are connected by non functional stream, but it does > not > > > > look > > > > >> > like > > > > >> > > a > > > > >> > > > > good > > > > >> > > > > >>>> option :). > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> > > > > >> > > > > >>>> On 1/25/16 08:52, Thomas Weise wrote: > > > > >> > > > > >>>> > > > > >> > > > > >>>> Sandeep suggested to connect them with a stream. > > > > >> > > > > >>>>> > > > > >> > > > > >>>>> On Mon, Jan 25, 2016 at 8:49 AM, Vlad Rozov < > > > > >> > > > [email protected] > > > > >> > > > > > > > > > >> > > > > >>>>> wrote: > > > > >> > > > > >>>>> > > > > >> > > > > >>>>> Sorry, possibly I miss something. X and Z are not > > > directly > > > > >> > > > connected > > > > >> > > > > >>>>> by a > > > > >> > > > > >>>>> > > > > >> > > > > >>>>>> stream, how can I apply THREAD_LOCAL? > > > > >> > > > > >>>>>> > > > > >> > > > > >>>>>> > > > > >> > > > > >>>>>> On 1/25/16 08:44, Thomas Weise wrote: > > > > >> > > > > >>>>>> > > > > >> > > > > >>>>>> Why does THREAD_LOCAL not work? > > > > >> > > > > >>>>>> > > > > >> > > > > >>>>>>> On Mon, Jan 25, 2016 at 7:43 AM, Vlad Rozov < > > > > >> > > > > [email protected] > > > > >> > > > > >>>>>>> > > > > > >> > > > > >>>>>>> wrote: > > > > >> > > > > >>>>>>> > > > > >> > > > > >>>>>>> Neither LOCALITY_HOST or THREAD_LOCAL provides > > > required > > > > >> > > > > >>>>>>> functionality. > > > > >> > > > > >>>>>>> The > > > > >> > > > > >>>>>>> > > > > >> > > > > >>>>>>> goal is to deploy X and Z to the same > > container/JVM. X > > > > >> may be > > > > >> > > > > >>>>>>>> deployed > > > > >> > > > > >>>>>>>> on > > > > >> > > > > >>>>>>>> any node. Z needs to be deployed on the same > > > container > > > > >> as X, > > > > >> > > so > > > > >> > > > X > > > > >> > > > > >>>>>>>> does > > > > >> > > > > >>>>>>>> not > > > > >> > > > > >>>>>>>> have any host affinity, and Z has affinity to X, > > not > > > to > > > > >> any > > > > >> > > > > specific > > > > >> > > > > >>>>>>>> host. > > > > >> > > > > >>>>>>>> If database client caches writes, reading inside > > the > > > > same > > > > >> > JVM > > > > >> > > > will > > > > >> > > > > >>>>>>>> be > > > > >> > > > > >>>>>>>> in > > > > >> > > > > >>>>>>>> process lookup, while deploying Z to any other > > > host/JVM > > > > >> > leads > > > > >> > > to > > > > >> > > > > >>>>>>>> inter > > > > >> > > > > >>>>>>>> process or inter host lookup. > > > > >> > > > > >>>>>>>> > > > > >> > > > > >>>>>>>> > > > > >> > > > > >>>>>>>> On 1/24/16 21:03, Thomas Weise wrote: > > > > >> > > > > >>>>>>>> > > > > >> > > > > >>>>>>>> There are the attributes LOCALITY_HOST and > > > > LOCALITY_RACK > > > > >> for > > > > >> > > an > > > > >> > > > > >>>>>>>> operator > > > > >> > > > > >>>>>>>> > > > > >> > > > > >>>>>>>> to > > > > >> > > > > >>>>>>>>> achieve precisely what you are describing. The > > value > > > > >> would > > > > >> > be > > > > >> > > > an > > > > >> > > > > >>>>>>>>> alias > > > > >> > > > > >>>>>>>>> that > > > > >> > > > > >>>>>>>>> can be mapped to physical hosts for allocation. > > The > > > > same > > > > >> > > could > > > > >> > > > be > > > > >> > > > > >>>>>>>>> used > > > > >> > > > > >>>>>>>>> of > > > > >> > > > > >>>>>>>>> anti-affinity, by assigning different values. > > > > >> > > > > >>>>>>>>> > > > > >> > > > > >>>>>>>>> Stream locality is a special case of expressing > > > > affinity > > > > >> > for > > > > >> > > 2 > > > > >> > > > > >>>>>>>>> operators. > > > > >> > > > > >>>>>>>>> Note that we also need a way to extend this to > > > > >> partitions > > > > >> > of > > > > >> > > > the > > > > >> > > > > >>>>>>>>> same > > > > >> > > > > >>>>>>>>> operator. > > > > >> > > > > >>>>>>>>> > > > > >> > > > > >>>>>>>>> > > > > >> > > > > >>>>>>>>> On Sun, Jan 24, 2016 at 8:52 PM, Vlad Rozov < > > > > >> > > > > >>>>>>>>> [email protected]> > > > > >> > > > > >>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>> > > > > >> > > > > >>>>>>>>> It may be quite extreme use case, but suppose > that > > > > >> > operator X > > > > >> > > > > >>>>>>>>> writes > > > > >> > > > > >>>>>>>>> to > > > > >> > > > > >>>>>>>>> a > > > > >> > > > > >>>>>>>>> > > > > >> > > > > >>>>>>>>> distributed in-memory database and operator Z > > reads > > > > from > > > > >> > it. > > > > >> > > Z > > > > >> > > > is > > > > >> > > > > >>>>>>>>> not > > > > >> > > > > >>>>>>>>> > > > > >> > > > > >>>>>>>>>> directly connected to X. In such cases it may > be > > > > >> necessary > > > > >> > > to > > > > >> > > > > let > > > > >> > > > > >>>>>>>>>> an > > > > >> > > > > >>>>>>>>>> application request container affinity > (deploying > > > to > > > > >> the > > > > >> > > same > > > > >> > > > > JVM) > > > > >> > > > > >>>>>>>>>> for > > > > >> > > > > >>>>>>>>>> operators X and Z as writes may be cached and > > reads > > > > >> from > > > > >> > the > > > > >> > > > > same > > > > >> > > > > >>>>>>>>>> JVM > > > > >> > > > > >>>>>>>>>> will > > > > >> > > > > >>>>>>>>>> be potentially faster. > > > > >> > > > > >>>>>>>>>> > > > > >> > > > > >>>>>>>>>> Technically the same may be applied to thread > > local > > > > on > > > > >> > NUMA > > > > >> > > > > >>>>>>>>>> boxes, as > > > > >> > > > > >>>>>>>>>> NUMA > > > > >> > > > > >>>>>>>>>> aware deployment of containers is not > supported. > > > > >> > > > > >>>>>>>>>> > > > > >> > > > > >>>>>>>>>> Vlad > > > > >> > > > > >>>>>>>>>> > > > > >> > > > > >>>>>>>>>> On 1/23/16 18:01, Yogi Devendra wrote: > > > > >> > > > > >>>>>>>>>> > > > > >> > > > > >>>>>>>>>> @Isha > > > > >> > > > > >>>>>>>>>> > > > > >> > > > > >>>>>>>>>> In my opinion: > > > > >> > > > > >>>>>>>>>> > > > > >> > > > > >>>>>>>>>>> THREAD_LOCAL, CONTAINER_LOCAL on stream is a > > > special > > > > >> case > > > > >> > > of > > > > >> > > > > >>>>>>>>>>> generic > > > > >> > > > > >>>>>>>>>>> rules > > > > >> > > > > >>>>>>>>>>> for Operator X and Operator Y. > > > > >> > > > > >>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>> We can say that, THREAD_LOCAL, CONTAINER_LOCAL > > > would > > > > >> be > > > > >> > > > > >>>>>>>>>>> applicable > > > > >> > > > > >>>>>>>>>>> only > > > > >> > > > > >>>>>>>>>>> if > > > > >> > > > > >>>>>>>>>>> operator X and Y are connected by stream. But, > > way > > > > to > > > > >> > > express > > > > >> > > > > >>>>>>>>>>> this > > > > >> > > > > >>>>>>>>>>> should > > > > >> > > > > >>>>>>>>>>> be similar to other rules for affinity. > > > > >> > > > > >>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>> ~ Yogi > > > > >> > > > > >>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>> On 24 January 2016 at 03:49, Isha Arkatkar < > > > > >> > > > > [email protected] > > > > >> > > > > >>>>>>>>>>> > > > > > >> > > > > >>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>> Hey Chinmay, > > > > >> > > > > >>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>> I certainly agree on common set of > rules > > > for > > > > >> > > > > configuring > > > > >> > > > > >>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>> affinity! > > > > >> > > > > >>>>>>>>>>>> Well > > > > >> > > > > >>>>>>>>>>>> put by a concrete example. :) > > > > >> > > > > >>>>>>>>>>>> Only thing I would like to point is: > > > > >> affinity of > > > > >> > > > > >>>>>>>>>>>> operators > > > > >> > > > > >>>>>>>>>>>> should > > > > >> > > > > >>>>>>>>>>>> not > > > > >> > > > > >>>>>>>>>>>> cover thread and container locality. Since > this > > > is > > > > >> only > > > > >> > > > > >>>>>>>>>>>> applicable > > > > >> > > > > >>>>>>>>>>>> when > > > > >> > > > > >>>>>>>>>>>> operators are connected by stream. So, it > makes > > > > >> sense to > > > > >> > > > have > > > > >> > > > > >>>>>>>>>>>> it on > > > > >> > > > > >>>>>>>>>>>> Stream > > > > >> > > > > >>>>>>>>>>>> rather than in common configuration. > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> And yes, DAG.validate should only > check > > > for > > > > >> > > REQUIRED > > > > >> > > > or > > > > >> > > > > >>>>>>>>>>>> STRICT > > > > >> > > > > >>>>>>>>>>>> policy. We > > > > >> > > > > >>>>>>>>>>>> can agree on one of the terminologies > > > > STRICT/RELAXED > > > > >> or > > > > >> > > > > >>>>>>>>>>>> REQUIRED/PREFERRED. > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> Thanks! > > > > >> > > > > >>>>>>>>>>>> Isha > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> On Fri, Jan 22, 2016 at 8:38 PM, Chinmay > > > Kolhatkar > > > > < > > > > >> > > > > >>>>>>>>>>>> [email protected]> > > > > >> > > > > >>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> Hi Isha, Bhupesh, > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> When I suggested singe affinity rule, I was > > > mainly > > > > >> > talking > > > > >> > > > > about > > > > >> > > > > >>>>>>>>>>>> "how > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> to" > > > > >> > > > > >>>>>>>>>>>>> of configuration and not of implementation. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> I see locality is in a way suggesting an > > > affinity > > > > of > > > > >> > > > > operators. > > > > >> > > > > >>>>>>>>>>>>> They're > > > > >> > > > > >>>>>>>>>>>>> close terminologies. > > > > >> > > > > >>>>>>>>>>>>> By configuring a locality on stream, we're > > also, > > > > in > > > > >> a > > > > >> > > way, > > > > >> > > > > >>>>>>>>>>>>> defining > > > > >> > > > > >>>>>>>>>>>>> affinity of operators. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> Until now, only locality was there and hence > > was > > > > >> > straight > > > > >> > > > > >>>>>>>>>>>>> forward > > > > >> > > > > >>>>>>>>>>>>> in > > > > >> > > > > >>>>>>>>>>>>> configuration for user. > > > > >> > > > > >>>>>>>>>>>>> Tomorrow, when anti-affinity configuration > > comes > > > > up, > > > > >> > one > > > > >> > > > > might > > > > >> > > > > >>>>>>>>>>>>> get > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> confused > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> on how to best use both locality & > > > anti-affinity. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> Hence suggested to make both > > (locality/affinity & > > > > >> > > > > >>>>>>>>>>>> anti-affinity) as > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> a > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> part > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> of single configuration. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> Suggestion is to have a more commonly adopted > > > > >> > > configuration > > > > >> > > > > >>>>>>>>>>>> which > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> admins > > > > >> > > > > >>>>>>>>>>>>> and developer's are familiar with. > > > > >> > > > > >>>>>>>>>>>>> Again referring to vShere Hypervisor's > > affinity > > > > >> rules. > > > > >> > I > > > > >> > > > > think > > > > >> > > > > >>>>>>>>>>>>> they > > > > >> > > > > >>>>>>>>>>>>> have > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> a > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> single configuration which does both. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> Having said that, here is a quick suggestion > on > > > how > > > > >> both > > > > >> > > can > > > > >> > > > > be > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> achieved > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> in > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> a single configuration: > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> CATEGORY TYPE POLICY > > > > >> ENTITIES > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> Affinity THREAD REQUIRED > > > > >> O1, O2 > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > >> > > > > >>>>>>>>>>>>> Operator1 & Operator2 should be thread local > > > > >> > > > > >>>>>>>>>>>>> Affinity NODE PREFERRED > > > > >> O3, O4 > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> O3 & > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> O4 are preferred to be in node local > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> AntiAffinity NODE REQUIRED > > > > O1, > > > > >> O4 > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> O1 & > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> O4 should not be on same node. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> AntiAffinity RACK PREFERRED > > > O2, > > > > O4 > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > >> > > > > >>>>>>>>>>>>> O2 > > > > >> > > > > >>>>>>>>>>>>> & > > > > >> > > > > >>>>>>>>>>>>> O4 are preferred not to be on same rack. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> Linux setting affinity of CPUs for threads > is > > > > >> another > > > > >> > way > > > > >> > > > of > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> configuration > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> we can take a look at. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> Learning from these commonly adopted > > > configuration > > > > >> > > pattern, > > > > >> > > > we > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> should > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> come > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> up with best configuration suitable for > > > > distributed > > > > >> > > > > >>>>>>>>>>>>> environment. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> Idea here is to not have our own > configuration > > > and > > > > >> give > > > > >> > > > > >>>>>>>>>>>> something > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> new > > > > >> > > > > >>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>> the users. Otherwise such an important > concept > > > > might > > > > >> > > > quickly > > > > >> > > > > >>>>>>>>>>>>> get > > > > >> > > > > >>>>>>>>>>>>> lost. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> Regarding the DAG.validate I think we would > > need > > > > to > > > > >> add > > > > >> > > > some > > > > >> > > > > >>>>>>>>>>>>> new > > > > >> > > > > >>>>>>>>>>>>> stuff > > > > >> > > > > >>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>> take care of anti-affinity. > > > > >> > > > > >>>>>>>>>>>>> Plus, anti-affinity/affinity should be > > validated > > > > at > > > > >> > > > > >>>>>>>>>>>>> DAG.validate > > > > >> > > > > >>>>>>>>>>>>> only > > > > >> > > > > >>>>>>>>>>>>> for > > > > >> > > > > >>>>>>>>>>>>> the ones which are required. > > > > >> > > > > >>>>>>>>>>>>> For preferred policies, validation in > logical > > > plan > > > > >> > might > > > > >> > > > be a > > > > >> > > > > >>>>>>>>>>>>> early > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> check. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> Thanks, > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>> Chinmay. > > > > >> > > > > >>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> On Sat, Jan 23, 2016 at 3:15 AM, Isha > > Arkatkar < > > > > >> > > > > >>>>>>>>>>>>> [email protected]> > > > > >> > > > > >>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> Hi, > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> Thanks for inputs! I like the idea of > > > > having > > > > >> > > single > > > > >> > > > > set > > > > >> > > > > >>>>>>>>>>>>> of > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> (anti?)affinity rules at dag context level. > > > > >> > > > > >>>>>>>>>>>>>> There could still be conflicts based > > on > > > > >> > Locality > > > > >> > > > for > > > > >> > > > > >>>>>>>>>>>>>> Streams > > > > >> > > > > >>>>>>>>>>>>>> or > > > > >> > > > > >>>>>>>>>>>>>> Node > > > > >> > > > > >>>>>>>>>>>>>> locality attribute set for operators. But > as > > > > >> Sandeep > > > > >> > > > > >>>>>>>>>>>>>> suggested, I > > > > >> > > > > >>>>>>>>>>>>>> also > > > > >> > > > > >>>>>>>>>>>>>> think Dag.validate should fail in case of > > > > >> > contradicting > > > > >> > > > > >>>>>>>>>>>>>> constraints. > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> We currently do not have 'affinity' > > among > > > > non > > > > >> > > stream > > > > >> > > > > >>>>>>>>>>>>>> operators, as > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> Bhupesh > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> pointed out. It is somewhat achievable by > > > > >> requesting > > > > >> > > node > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> locality > > > > >> > > > > >>>>>>>>>>>>> for > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> operators (Assuming node requests work as > > > > expected). > > > > >> > But > > > > >> > > > > >>>>>>>>>>>>> should we > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> consider > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> adding affinity specifications support as > > well > > > > >> along > > > > >> > > with > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> anti-affinity? > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> Regarding specification of attributes > > > from > > > > >> > > > > dt-site.xml. > > > > >> > > > > >>>>>>>>>>>>>> We > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> can > > > > >> > > > > >>>>>>>>>>>>> go > > > > >> > > > > >>>>>>>>>>>>> with > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> Json like string or even xml representation > > for > > > > >> complex > > > > >> > > > > >>>>>>>>>>>>> objects. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> What > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> is > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> our current behavior for setting Java > object > > > > >> > properties > > > > >> > > > > >>>>>>>>>>>>>> through > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> XML? > > > > >> > > > > >>>>>>>>>>>>> We > > > > >> > > > > >>>>>>>>>>>>> can > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> follow the same for this as well. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> As for precedence or ability to > satisfy > > > > >> > > constraints: > > > > >> > > > > >>>>>>>>>>>>> Right > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> now in > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> normal > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> scenario, if resources are not available > for > > > > >> > allocating > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> containers, > > > > >> > > > > >>>>>>>>>>>>> we > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> keep > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> sending to request till all are obtained. > > > > >> Likewise, in > > > > >> > > > case > > > > >> > > > > of > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> strict > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> anti-affinity policy, we should keep the > > > > >> application in > > > > >> > > > > >>>>>>>>>>>>> ACCEPTED > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> state > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> till > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> the anti-affinity constraint is satisfied. > > For > > > > >> relaxed > > > > >> > > > > >>>>>>>>>>>>>> policy, we > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> can > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> decide timeout for relaxing the > anti-affinity > > > > rule. > > > > >> > > Please > > > > >> > > > > note > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> this > > > > >> > > > > >>>>>>>>>>>>>> applies only when we have non-contradicting > > > > rules. > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> Thanks! > > > > >> > > > > >>>>>>>>>>>>>> Isha > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 5:30 AM, Bhupesh > > > Chawda < > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> [email protected] > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> I agree on having a single set of rules for > > the > > > > >> > affinity > > > > >> > > as > > > > >> > > > > >>>>>>>>>>>>> well > > > > >> > > > > >>>>>>>>>>>>> as > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> anti-affinity of operators / partitions on > > > > >> containers. > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> However, I noted the following points: > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> 1. AFAIK, we do not support > affinity > > > > >> > (locality) > > > > >> > > > in > > > > >> > > > > a > > > > >> > > > > >>>>>>>>>>>>>>> general > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> sense. > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> The > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> affinity is only for a stream, not > > for > > > > >> *any* > > > > >> > two > > > > >> > > > > >>>>>>>>>>>>> operators. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> So, > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> we > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> should also look at the general > case > > > and > > > > >> see > > > > >> > > how > > > > >> > > > it > > > > >> > > > > >>>>>>>>>>>>>>> can > > > > >> > > > > >>>>>>>>>>>>>>> be > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> supported, > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> if > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> there are valid use cases. > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> 2. Coming to anti-affinity, we > cannot > > > > type > > > > >> > cast > > > > >> > > it > > > > >> > > > > as > > > > >> > > > > >>>>>>>>>>>>>> a > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> type of > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> affinity > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> rule. Saying "two operators must > be > > on > > > > the > > > > >> > same > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> container" is > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> very > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> different from saying "these two > > > > operators > > > > >> > must > > > > >> > > > not > > > > >> > > > > >>>>>>>>>>>>>>> be on > > > > >> > > > > >>>>>>>>>>>>>>> the > > > > >> > > > > >>>>>>>>>>>>>>> same > > > > >> > > > > >>>>>>>>>>>>>>> container". In this case, the > second > > > one > > > > >> is a > > > > >> > > > much > > > > >> > > > > >>>>>>>>>>>>>>> more > > > > >> > > > > >>>>>>>>>>>>>>> relaxed > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> rule > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> as > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> compared to the first one. > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> 3. Once we have this generic set of > > > rules, > > > > >> > there > > > > >> > > > must > > > > >> > > > > >>>>>>>>>>>>>> be a > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> "satisfiability" test run before > > > > >> requesting > > > > >> > > YARN > > > > >> > > > > for > > > > >> > > > > >>>>>>>>>>>>>>> containers. > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> If > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> the > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> request is not satisfiable, then > > there > > > is > > > > >> no > > > > >> > > point > > > > >> > > > > >>>>>>>>>>>>> asking > > > > >> > > > > >>>>>>>>>>>>> YARN > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> allocate > > > > >> > > > > >>>>>>>>>>>>>>> containers in this manner. In case > > it > > > is > > > > >> not > > > > >> > > > > >>>>>>>>>>>>>>> satisfiable, we > > > > >> > > > > >>>>>>>>>>>>>>> must > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> also > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> have > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> a default order in which the rules > > can > > > be > > > > >> > > > "relaxed" > > > > >> > > > > >>>>>>>>>>>>>> and > > > > >> > > > > >>>>>>>>>>>>>> the > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> request > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> be > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> made > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> satisfiable. For example, some very > > > strict > > > > >> > rules > > > > >> > > > may > > > > >> > > > > be > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> ignored, > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> or > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> made > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> less constraining ( for example "on > > the > > > > >> same > > > > >> > > > > >>>>>>>>>>>>> container" => > > > > >> > > > > >>>>>>>>>>>>> "on > > > > >> > > > > >>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>> the > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> same > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> node"). > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> Thanks. > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> -Bhupesh > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:54 PM, Aniruddha > > > > >> Thombare < > > > > >> > > > > >>>>>>>>>>>>>>> [email protected]> wrote: > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> +1 On Chinmay's suggestion about having > > single > > > > >> set of > > > > >> > > > > >>>>>>>>>>>>>>> affinity > > > > >> > > > > >>>>>>>>>>>>>>> rules. > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> Thanks, > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> Aniruddha > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:57 PM, Sandeep > > > > >> Deshmukh < > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> [email protected] > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> Between 2 operators, if one configures > > > > >> > thread/container > > > > >> > > > > local > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> and > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> anti-affinity as well, which one will > take > > > > >> affect? > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> The DAG validation step should error out > > in > > > > this > > > > >> > > case. > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> +1 on suggestion by Chinmay to name it > > > > "Affinity > > > > >> > > Rules" > > > > >> > > > > >>>>>>>>>>>>>>> than > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> anti-affinity. We are just extending our > > > > container > > > > >> > > > > allocation > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> scheme > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> support containers not to be allocated > > > > together. > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> Regards, > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> Sandeep > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:43 PM, Chinmay > > > > >> Kolhatkar > > > > >> > < > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]> > > > > >> > > > > >>>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> Hi Isha, > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> Couple of points: > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> 1. About the interface to configuring > > > > >> > anti-affinity, > > > > >> > > as > > > > >> > > > > per > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> suggestion > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> above there are 2 different way to > > > configure > > > > >> > > locality > > > > >> > > > > and > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> anti-affinity: > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> i.e. dag.setAttribute - for anti-affinity > > & > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> dag.addStream(...).setLocality for > > > > >> locality. > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> Between 2 operators, if one configures > > > > >> > > thread/container > > > > >> > > > > >>>>>>>>>>>>>>>>>> local > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> and > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity as well, which one will > > take > > > > >> affect? > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> 2. Consider there could be such > confusion > > as > > > > >> above, > > > > >> > > > would > > > > >> > > > > >>>>>>>>>>>>>>>> it > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> make > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> sense > > > > >> > > > > >>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> have a single API which takes care of both > > > > >> > > anti-affinity > > > > >> > > > > and > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> locality. > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> This > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> way, one is configurable. > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> 3. This point is coming from how VM > > affinity > > > is > > > > >> > > > configured > > > > >> > > > > >>>>>>>>>>>>>>>>> in > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> vSphere. > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> The VMs are configured affinity are > > called > > > as > > > > >> > > > "affinity > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> rules" > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> and > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> not > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> "anti-affinity rules". Ultimately idea > is > > to > > > > >> > allocate > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> processing > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> nodes. > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> Via "VM-VM affinity rules", anti-affinity > is > > > > also > > > > >> > > > > configured. > > > > >> > > > > >>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>> But > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> there > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> is > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> a single set of rule definition for both > > > > affinity > > > > >> > > > (similar > > > > >> > > > > to > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> locality > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> in > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> our case) and anti-affinity. > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> Would it be a better approach for > > configuring > > > > >> > locality > > > > >> > > > > rules > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> and > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity rules in a single rule > and > > > call > > > > >> it > > > > >> > > > > "affinity > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> rule". > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> Thanks, > > > > >> > > > > >>>>>>>>>>>>>>> Chinmay. > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 12:24 PM, Yogi > > > Devendra > > > > < > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected] > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> @Isha > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> I understand that anti-affinity across > > > > >> application > > > > >> > is > > > > >> > > > not > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> straight-forward. > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> It would be OK even if we do not have > it > > > in > > > > >> > > iteration > > > > >> > > > > 1. > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> But, for attributes syntax; I still > think > > > > that > > > > >> > Java > > > > >> > > > > object > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> should > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> be > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> avoided as they will be hard to > configure > > > > from > > > > >> > > > > >>>>>>>>>>>>>>>>> dt-site.xml or > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> other > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> config > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> files. > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> Other suggestion for this could be JSON > > > > >> > representation > > > > >> > > > of > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> String > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> array: > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> ["O2", "O3"]. (If operator names has > > some > > > > >> special > > > > >> > > > > >>>>>>>>>>>>>>>>> characters > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> like > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> " > > > > >> > > > > >>>>>>>>>>>>>>>>> or [ > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> or , those will be escaped in the JSON > > > > >> > > representation.) > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> Not sure if others agree on this; but > > > attribute > > > > >> > syntax > > > > >> > > > > >>>>>>>>>>>>>>>>> should > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> be > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> finalized > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> in iteration 1 to avoid backward > > > > compatibility > > > > >> > > issues > > > > >> > > > > >>>>>>>>>>>>>>>>> later. > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> ~ Yogi > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> On 22 January 2016 at 00:43, Thomas > Weise < > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected]> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> Node based requests is the best > approach > > - > > > if > > > > >> it > > > > >> > > works > > > > >> > > > > :-) > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> Blacklisting will require to allocate the > > > > >> containers > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> sequentially. > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> It > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> will > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> work, but slow down application startup, > > > > >> especially > > > > >> > > for > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> larger > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> topologies. > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jan 21, 2016 at 10:42 AM, Isha > > > > >> Arkatkar < > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]> > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> Hi, > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> Should we consider the node > based > > > > >> requests > > > > >> > if > > > > >> > > > it > > > > >> > > > > >>>>>>>>>>>>>>>>>>> works > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> with > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Capacity > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> Scheduler or avoid 2b approach > > > altogether? > > > > I > > > > >> > > checked > > > > >> > > > > >>>>>>>>>>>>>>>>>>> that > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> node > > > > >> > > > > >>>>>>>>>>>>>>>>> requests > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> do > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> not work with fair scheduler on CDH > > > cluster. > > > > >> Yarn > > > > >> > > does > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> not > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> return > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> any > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> container if hostname is given in the > > > > container > > > > >> > > > request. > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> I > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> am > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> trying > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> setup a small virtual hortonworks cluster > > to > > > > >> check > > > > >> > the > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> this > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> behavior > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> on > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> that. > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> YARN-2027 < > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > https://issues.apache.org/jira/browse/YARN-2027 > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> mentioned > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> that > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> container requests are not honored in > > > > capacity > > > > >> > > > scheduler > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> too. > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> But I > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> am > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> not > > > > >> > > > > >>>>>>>>>>>>>>>>> sure if it is because of distro > dependent > > > > issue. > > > > >> > > Please > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> share > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> insights. > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> @Vlad, Adding support for regular > > > > expression > > > > >> > > sounds > > > > >> > > > > >>>>>>>>>>>>>>>>>>> good. > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> We > > > > >> > > > > >>>>>>>>>>>>>>>>> could > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> translate to list of operator names > > > > internally > > > > >> > based > > > > >> > > > on > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> regex. > > > > >> > > > > >>>>>>>>>>>>>>>>> @Yogi, I went with a list of strings > for > > > > >> attribute > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> because > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> "O2, > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> O3" > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> could > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> be a valid single operator name too :) > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> I am not sure of ways to implement > > > > anti-affinity > > > > >> > > across > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> application. > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Though > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> something to consider for later > > > iteration. > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks, > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> Isha > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:59 PM, > > Thomas > > > > >> Weise < > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> https://issues.apache.org/jira/browse/SLIDER-82 > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:56 PM, > Thomas > > > > Weise > > > > >> < > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> The point was that containers are > > taken > > > > away > > > > >> > from > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> other > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> apps > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> that > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> may > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> have > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> to discard work etc. It's not good style > > to > > > > >> claim > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> resources > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> and > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> not > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> use > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> them eventually :-) > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> For this feature it is necessary to > look > > > at > > > > >> the > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> scheduler > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> capabilities/semantics and > > > limitations. > > > > >> For > > > > >> > > > > example, > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> don't > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> bet > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> exclusively > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> on node requests if the goal is for it > to > > > > work > > > > >> > with > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> FairScheduler. > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Also look at Slider, which just > recently > > > > added > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> support > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> for > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> anti-affinity > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> (using node requests). When you run > it > > on > > > > the > > > > >> > CDH > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> cluster, > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> it > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> probably > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> won't work... > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 3:19 PM, Pramod > > > > >> Immaneni < > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected] > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Once released won't the containers > be > > > > >> > available > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> again > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> in > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> the > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> pool. > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> This > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> would only be optional and not mandatory. > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> Thanks > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at 2:02 PM, > Thomas > > > > >> Weise < > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected] > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> How about also supporting a minor > > > > >> variation > > > > >> > of > > > > >> > > it > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> as > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> an > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> option > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> where it greedily gets the total > > number > > > > of > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> containers > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> and > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> discards > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ones > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> it > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> can't use and repeats the process for > > the > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> remaining > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> till > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> everything > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> has > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> been allocated. > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> This is problematic as with resource > > > > preemption > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> these > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> containers > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> will > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> potentially taken away from other > > > > >> applications > > > > >> > and > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> then > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> thrown > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> away. > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Also does it make sense to support > > > > >> > anti-cluster > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> affinity? > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at 1:21 PM, > > > Isha > > > > >> > > Arkatkar > > > > >> > > > < > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected]> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Hi all, > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> We want add support for > > > > >> > Anti-affinity > > > > >> > > in > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Apex > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> allow > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> applications > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> launch specific physical operators on > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> different > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> nodes(APEXCORE-10 > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> < > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > https://issues.apache.org/jira/browse/APEXCORE-10 > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ). > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> Want > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> request > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> your > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> suggestions/ideas for the same! > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> The reasons for using > > > > anti-affinity > > > > >> in > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> could > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> be: > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ensure > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> reliability, for performance reasons > > (such > > > as > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> application > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> may > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> not > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> want > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> 2 > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> i/o intensive operators to land on > > the > > > > same > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> node > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> improve > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> performance) > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> or > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> for some application specific > > > constraints(for > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> example, > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> 2 > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> partitions > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> cannot > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be run on the same node since they > > use > > > > same > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> port > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> number). > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> This > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> is > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> general rationale for adding > Anti-affinity > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> support. > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Since, Yarn does not support > > > > >> anti-affinity > > > > >> > yet > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> (YARN-1042 > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> < > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > https://issues.apache.org/jira/browse/YARN-1042 > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> ), > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> we > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> need > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> implement > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>> the logic in AM. Wanted to get your > > views > > > on > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> following > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> aspects > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> for > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> this > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> implementation: > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> *1. How to specify anti-affinity for > > > > physical > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators/partitions > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> in > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> application:* > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> One way for this is to > > have > > > an > > > > >> > > > attribute > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> for > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> setting > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> at > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the logical operator context. And > an > > > > >> operator > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> can > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> set > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> this > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> attribute > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> with > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> list of operator names which should not > > be > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> collocated. > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Consider dag with 3 > > > operators: > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> TestOperator o1 = > > > > >> > > > > dag.addOperator("O1", > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> TestOperator o2 = > > > > >> > dag.addOperator("O2", > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> TestOperator o3 = > > > > >> dag.addOperator("O3", > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> To set anti-affinity for O1 > > operator: > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> dag.setAttribute(o1, > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> OperatorContext.ANTI_AFFINITY, > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>> new > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > ArrayList<String>(Arrays.asList("O2", > > > > >> > > "O3"))); > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> This would mean O1 > should > > > not > > > > >> be > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> allocated > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> on > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> nodes > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> containing > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> operators O2 and O3. This applies > to > > > all > > > > >> > > > > >>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>> allocated > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> partitions > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> of > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> O1, > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> O2, > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>> O3. > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> Also, if same operator > name > > is > > > > >> part > > > > >> > of > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> list, > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > >> > > > > >> > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > >
