Hi Isha, Couple of points: 1. Do the first curly brackets required? They seem to add another level of hierarchy. 2. operatorRegex, operators, operatorList all seem to mention list list of operators. Do you think we can come up with a simpler rather than providing more options? 3. Having JSON innside xml seems a little convoluted to me. I wonder if there is a better way to set dt.attr.AFFINITY_RULES_SET. Maybe have t as XML itself? 4. For operatorRegex which regex language do we support? Is it Java regex OR perl/bash based one?
Thanks, Chinmay. On Sat, Feb 27, 2016 at 3:38 AM, Isha Arkatkar <[email protected]> wrote: > Hi, > > To support setting affinity rules from properties.xml, I added json > string codec to read the value. I had to wrap the list of rules in an > object call AffinityRuleSet to support easy serialization. > > Here is an example for setting through property: > > <property> > <name>dt.attr.AFFINITY_RULES_SET</name> > <value> > { > "affinityRules": [ > { > "operatorRegex": "rand|console", > "locality": "NODE_LOCAL", > "type": "AFFINITY", > "relaxLocality": false > }, > { > "operators": { > "first": "rand", > "second": "passThru" > }, > "locality": "NODE_LOCAL", > "type": "ANTI_AFFINITY", > "relaxLocality": false > }, > { > "operatorsList": [ > "passThru", > "passThru" > ], > "locality": "NODE_LOCAL", > "type": "ANTI_AFFINITY", > "relaxLocality": false > } > ] > } > </value> > </property> > > Thanks, > Isha > > On Thu, Feb 25, 2016 at 8:16 PM, Chinmay Kolhatkar <[email protected]> > wrote: > > > Hi Isha, > > > > 2 ways looks fine. > > > > Just one point, how would user set affinity rules using DAG content > > attribute from properties.xml file? > > Can you please provide some example for this? > > > > Thanks, > > Chinmay. > > > > > > On Fri, Feb 26, 2016 at 2:57 AM, Isha Arkatkar <[email protected]> > > wrote: > > > > > Thanks for the input Chinmay. > > > > > > The last parameter true/false is for relaxing the constraint. If true, > > the > > > affinity rule would be relaxed if constraint cannot be satisfied. And > if > > > false, application will keep requesting and waiting to launch > containers > > > till all rules are satisfied. > > > > > > Regarding the dag.setAffinity APIs. I can add those couple APIs in > > > LogicalPlan as a wrapper that internally can set the Dag Attributes. > > > Probably yes, that would be easier to use, to set affinity directly on > > dag > > > object. > > > > > > Then there would 2 ways to set affinity rules: > > > > > > 1. Through DAG context attribute (Either through code or through > > > properties.xml) > > > 2. By calling setAffinity on dag. > > > > > > Thanks, > > > Isha > > > > > > On Wed, Feb 24, 2016 at 10:11 PM, Chinmay Kolhatkar < > [email protected]> > > > wrote: > > > > > > > I agree with Pramod. We should probably have single interface for > > > > specifying both if at all possible. This is in interest of not > > confusing > > > > users with two different configurations at the same time make a > > > > locality/affinity API to apear similar to what is out there in the > word > > > at > > > > the moment. > > > > > > > > Though it might be early to deprecate stream locality, but after some > > > time > > > > based on user's experience we should consider deprecating Stream > > Locality > > > > API. > > > > > > > > @Isha, nice APIs. Questions/suggestions regarding this: > > > > 1. What does the "false" signify for AffinityRule constructor. > > > > 2. This is a suggestion. Instead of having a List<AffinityRule> and > > then > > > > setting this object to attribute of the DAG, should we have an API in > > > > LogicalPlan like following: > > > > > > > > dag.setAffinity(<operator1>, <operator2>, <locality_type>, > false/true) > > > > > > > > dag.setAntiAffinity(<operator1>, <operator2>, false/true) // Not > sure > > > > what false/true mean, added for completion. But the point here is to > > > remove > > > > locality for anti-affinity. > > > > > > > > > > > > Thanks, > > > > Chinmay. > > > > > > > > > > > > > > > > On Thu, Feb 25, 2016 at 5:17 AM, Pramod Immaneni < > > [email protected] > > > > > > > > wrote: > > > > > > > > > You could specify it by specifying the names of all operators that > > are > > > > part > > > > > of the stream. > > > > > > > > > > On Wed, Feb 24, 2016 at 3:44 PM, Amol Kekre <[email protected]> > > > > wrote: > > > > > > > > > > > These are two diff ways of specifying something similar, but not > > > always > > > > > > same. For example, how do I specify the following > > > > > > > > > > > > This stream has way to high throughput. Force all operators that > > > > connect > > > > > to > > > > > > it today (or in future) to be container local. > > > > > > > > > > > > With stream locality I get that as operator ports are added to > the > > > > > stream, > > > > > > they automatically inherit stream locality. > > > > > > > > > > > > Thks, > > > > > > Amol > > > > > > > > > > > > On Wed, Feb 24, 2016 at 3:37 PM, Pramod Immaneni < > > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > To explain the affinity support below seems to be a super-set > > that > > > > can > > > > > > > consume the stream locality setting whether that was the > original > > > > > > intention > > > > > > > or not. > > > > > > > > > > > > > > On Wed, Feb 24, 2016 at 3:35 PM, Pramod Immaneni < > > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Why would we want two different ways of specifying the same > > > thing? > > > > > > > > > > > > > > > > On Wed, Feb 24, 2016 at 3:29 PM, Amol Kekre < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > >> Stream locality is a stream attribute. Affinity is operator > > > > > attribute. > > > > > > > >> Technically they may not be connected. Though operator > > affinity > > > > can > > > > > be > > > > > > > >> extended to specify a situation that implicity covers stream > > > > > locality, > > > > > > > it > > > > > > > >> may be better to let user say the two following statements > > > > > > > independently. > > > > > > > >> > > > > > > > >> - This stream is heavy on I/O and I want it to be ..._local > // > > > > This > > > > > > > >> statement is independent of future additions of operators to > > > this > > > > > > stream > > > > > > > >> - I need these logical operators to not be (or be) on the > same > > > > node > > > > > > > >> > > > > > > > >> Do note that container_local or node_local stream localities > > > only > > > > > > relate > > > > > > > >> to > > > > > > > >> connected operators. > > > > > > > >> > > > > > > > >> Thks > > > > > > > >> Amol > > > > > > > >> > > > > > > > >> > > > > > > > >> On Wed, Feb 24, 2016 at 1:54 PM, Pramod Immaneni < > > > > > > > [email protected]> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > Would it make sense to deprecate stream locality in favor > of > > > > this? > > > > > > > >> > > > > > > > > >> > On Tue, Feb 23, 2016 at 6:05 PM, Isha Arkatkar < > > > > > > [email protected]> > > > > > > > >> > wrote: > > > > > > > >> > > > > > > > > >> > > Hi Pramod, > > > > > > > >> > > > > > > > > > >> > > We can have a list or operators or regex to specify > > > > operators > > > > > > > list > > > > > > > >> for > > > > > > > >> > > affinity rule. Though, internal implementation will > > > translate > > > > > both > > > > > > > >> these > > > > > > > >> > > into list of pairs. Since, having pair of operators > helps > > in > > > > > > > >> validation > > > > > > > >> > > phase. > > > > > > > >> > > > > > > > > > >> > > And yes, validation phase will catch conflicting > locality > > > or > > > > > > > affinity > > > > > > > >> > > rules and throw validation exception. > > > > > > > >> > > > > > > > > > >> > > Thanks for input regarding unifiers. It is probably > > alright > > > > to > > > > > > not > > > > > > > >> > include > > > > > > > >> > > it in the affinity settings. Will leave the current > > > > > implementation > > > > > > > as > > > > > > > >> is. > > > > > > > >> > > > > > > > > > >> > > Thanks! > > > > > > > >> > > Isha > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > On Tue, Feb 23, 2016 at 5:56 PM, Pramod Immaneni < > > > > > > > >> [email protected] > > > > > > > >> > > > > > > > > > >> > > wrote: > > > > > > > >> > > > > > > > > > >> > > > Isha, > > > > > > > >> > > > > > > > > > > >> > > > If the implementation would support (eventually) > > affinity > > > > > > between > > > > > > > >> more > > > > > > > >> > > than > > > > > > > >> > > > 2 logical operators, instead of relying on specifying > it > > > as > > > > > > > multiple > > > > > > > >> > > rules > > > > > > > >> > > > why not make provision in the API to specify more > than 2 > > > > > logical > > > > > > > >> > > operators? > > > > > > > >> > > > If there is a stream locality rule that conflicts with > > the > > > > > > > affinity > > > > > > > >> > rule > > > > > > > >> > > > will the validation catch this? With unifiers the > common > > > > case > > > > > of > > > > > > > MxN > > > > > > > >> > > > locality is already chosen based on downstream > operator. > > > In > > > > > > other > > > > > > > >> cases > > > > > > > >> > > > such as single downstream or cascade case I am not > sure > > it > > > > is > > > > > > all > > > > > > > >> the > > > > > > > >> > > > important to have it. > > > > > > > >> > > > > > > > > > > >> > > > Thanks > > > > > > > >> > > > > > > > > > > >> > > > On Tue, Feb 23, 2016 at 5:37 PM, Isha Arkatkar < > > > > > > > >> [email protected]> > > > > > > > >> > > > wrote: > > > > > > > >> > > > > > > > > > > >> > > > > Hi all, > > > > > > > >> > > > > > > > > > > > >> > > > > I have opened a review only pull request for > > > handling > > > > > > > affinity > > > > > > > >> > > rules > > > > > > > >> > > > in > > > > > > > >> > > > > Apex: > > > > > > > >> > > > > > > > https://github.com/apache/incubator-apex-core/pull/234 > > > > > > > >> > > > > > > > > > > > >> > > > > Just wanted to confirm with everyone, if the APIs > for > > > > > > > specifying > > > > > > > >> > > > affinity > > > > > > > >> > > > > rules look alright. Also, wanted to get views on how > > > > > unifiers > > > > > > > >> should > > > > > > > >> > be > > > > > > > >> > > > > handled in case of operator affinity rules. More > > details > > > > > > follow: > > > > > > > >> > > > > > > > > > > > >> > > > > I have added a list of affinity rules as an > attribute > > > in > > > > > Dag > > > > > > > >> > context. > > > > > > > >> > > > For > > > > > > > >> > > > > now, have added AffinityRule to be specified for > > > operator > > > > > > pairs. > > > > > > > >> We > > > > > > > >> > can > > > > > > > >> > > > add > > > > > > > >> > > > > regex support in next iteration. > > > > > > > >> > > > > > > > > > > > >> > > > > Here is sample usage: > > > > > > > >> > > > > > > > > > > > >> > > > > List<AffinityRule> rules = new ArrayList<>(); > > > > > > > >> > > > > // To add node locality between 2 not connected > > > > > operators > > > > > > > rand > > > > > > > >> > and > > > > > > > >> > > > > console: > > > > > > > >> > > > > rules.add(new AffinityRule(Type.AFFINITY, new > > > > > OperatorPair( > > > > > > > >> > "rand", > > > > > > > >> > > > > "console"), Locality.NODE_LOCAL, false)); > > > > > > > >> > > > > > > > > > > > >> > > > > // To add anti-affinity between all partitions of > > > rand > > > > > > > operator > > > > > > > >> > with > > > > > > > >> > > > all > > > > > > > >> > > > > partitions of passThru operator: > > > > > > > >> > > > > rules.add(new AffinityRule(Type.ANTI_AFFINITY, > new > > > > > > > >> OperatorPair( > > > > > > > >> > > > "rand", > > > > > > > >> > > > > "passThru"), Locality.NODE_LOCAL, false)); > > > > > > > >> > > > > > > > > > > > >> > > > > // To add anti-affinity between partitions of > > > passThru > > > > > > > >> operator, > > > > > > > >> > > give > > > > > > > >> > > > > same operator name in pair: > > > > > > > >> > > > > rules.add(new AffinityRule(Type.ANTI_AFFINITY, > new > > > > > > > >> OperatorPair( > > > > > > > >> > > > > "passThru", "passThru"), Locality.NODE_LOCAL, > false)); > > > > > > > >> > > > > > > > > > > > >> > > > > // Set the rules in dag context > > > > > > > >> > > > > dag.setAttribute(DAGContext.AFFINITY_RULES, > rules); > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > Please find a sample application using these > affinity > > > > rules > > > > > > > here: > > > > > > > >> > > > > > > > > > > > >> > > > > https://github.com/ishark/Apex-Samples/tree/master/affinity-example > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > The actual implementation of affinity rules heavily > > > > depends > > > > > > node > > > > > > > >> > > specific > > > > > > > >> > > > > requests, that was already being handled in > > > > > > > >> > StreamingAppMasterService. > > > > > > > >> > > To > > > > > > > >> > > > > handle node requests for cloudera, I have added an > > > > override > > > > > to > > > > > > > >> > > blacklist > > > > > > > >> > > > > all other nodes except the ones where container > > request > > > is > > > > > to > > > > > > be > > > > > > > >> > > issued. > > > > > > > >> > > > > > > > > > > > >> > > > > There is one open question I wanted to bring up: In > > case > > > > of > > > > > > > >> > partitioned > > > > > > > >> > > > > operators, should the unifiers also follow affinity > or > > > > > > > >> anti-affinity > > > > > > > >> > > > rules? > > > > > > > >> > > > > Or should they be independent. For now, I have kept > > them > > > > > > > >> independent > > > > > > > >> > > and > > > > > > > >> > > > > only the actual operators follow affinity rules. > > Please > > > > > > suggest > > > > > > > >> what > > > > > > > >> > > > would > > > > > > > >> > > > > make more sense from user's specification > perspective. > > > > > > > >> > > > > > > > > > > > >> > > > > Thanks, > > > > > > > >> > > > > Isha > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > On Mon, Feb 1, 2016 at 2:12 PM, Isha Arkatkar < > > > > > > > >> [email protected]> > > > > > > > >> > > > > wrote: > > > > > > > >> > > > > > > > > > > > >> > > > > > Hi folks, > > > > > > > >> > > > > > > > > > > > > >> > > > > > Summarizing the proposal for > > affinity/anti-affinity > > > > > rules > > > > > > > in > > > > > > > >> > apex > > > > > > > >> > > as > > > > > > > >> > > > > > per discussions in this mail thread. Please > suggest > > > if I > > > > > > > missed > > > > > > > >> > > > > something. > > > > > > > >> > > > > > > > > > > > > >> > > > > > * For configuration:* > > > > > > > >> > > > > > > > > > > > > >> > > > > > - We will have application level > > > > affinity/anti-affinity > > > > > > > rules. > > > > > > > >> > For > > > > > > > >> > > > the > > > > > > > >> > > > > > first iteration, we can support for > (anti-)affinity > > > > among > > > > > > > >> operators > > > > > > > >> > > > > within > > > > > > > >> > > > > > single application. We can revisit it in the next > > > > > iteration > > > > > > to > > > > > > > >> > > support > > > > > > > >> > > > > > across applications. > > > > > > > >> > > > > > > > > > > > > >> > > > > > - Each rule will consist of 4 components: > > > > > > > >> > > > > > <List of operators>, AFFINITY/ANTI-AFFINITY, > > > > > > > >> STRICT/RELAXED > > > > > > > >> > > > > policy, > > > > > > > >> > > > > > CONTAINER/NODE/RACK > > > > > > > >> > > > > > I have checked that Apex supports container > locality > > > > > > between 2 > > > > > > > >> > > > operators > > > > > > > >> > > > > > that are not connected by Steam. Though did not > find > > > an > > > > > API > > > > > > > for > > > > > > > >> > > setting > > > > > > > >> > > > > it > > > > > > > >> > > > > > through application. So, affinity rules can cover > > > > > container > > > > > > > >> > affinity > > > > > > > >> > > as > > > > > > > >> > > > > > well. > > > > > > > >> > > > > > We will not support THREAD locality in app level > > > > affinity > > > > > or > > > > > > > >> > > > > > anti-affinity, since it does not apply to > > unconnected > > > > > > > operators. > > > > > > > >> > > > > > > > > > > > > >> > > > > > The list of operators can also be specified by > > regex > > > > > > > >> > > > > > > > > > > > > >> > > > > > *For Implementation: * > > > > > > > >> > > > > > > > > > > > > >> > > > > > - I have verified that Node specific requests > are > > > > > honored > > > > > > by > > > > > > > >> > > > > HortonWorks > > > > > > > >> > > > > > for both Fair and Capacity scheduler. So we can > go > > > with > > > > > the > > > > > > > >> > approach > > > > > > > >> > > > of > > > > > > > >> > > > > > node specific container requests to implement > > > affinity > > > > > > rules. > > > > > > > >> > > > > > > > > > > > > >> > > > > > - However, we may need to fall back on blacklist > > > > approach > > > > > > for > > > > > > > >> CDH > > > > > > > >> > > > > distro, > > > > > > > >> > > > > > as node specific requests may not work well. As: > > > > > > > >> > > > > > -> With Fair scheduler, node specific > > > requests > > > > > are > > > > > > > not > > > > > > > >> > > honored > > > > > > > >> > > > > in > > > > > > > >> > > > > > CDH 4. Application master keeps sending requests > to > > > > Yarn, > > > > > > but > > > > > > > no > > > > > > > >> > > > > containers > > > > > > > >> > > > > > are allocated. > > > > > > > >> > > > > > -> I could not yet verify the same with > > Fair > > > > and > > > > > > > >> Capacity > > > > > > > >> > > > > > Scheduler on CDH 5. Ran into some configuration > > issues > > > > > when > > > > > > > >> setting > > > > > > > >> > > up > > > > > > > >> > > > > CDH > > > > > > > >> > > > > > 5.5.1. Will update the thread with findings. > > > > > > > >> > > > > > > > > > > > > >> > > > > > I will try to build a prototype for affinity > rules > > > > with > > > > > > > these > > > > > > > >> > > things > > > > > > > >> > > > in > > > > > > > >> > > > > > mind. Please let me know if anyone would like to > > > > > collaborate > > > > > > > on > > > > > > > >> > this > > > > > > > >> > > > > > effort! :) > > > > > > > >> > > > > > > > > > > > > >> > > > > > Thanks! > > > > > > > >> > > > > > Isha > > > > > > > >> > > > > > > > > > > > > >> > > > > > On Mon, Jan 25, 2016 at 10:23 AM, Vlad Rozov < > > > > > > > >> > > [email protected]> > > > > > > > >> > > > > > wrote: > > > > > > > >> > > > > > > > > > > > > >> > > > > >> Yes, with ability to refer not only to a host > name, > > > but > > > > > to > > > > > > > >> > operators > > > > > > > >> > > > > >> (possibly both in the same and other > applications). > > > In > > > > > > > general > > > > > > > >> > this > > > > > > > >> > > > > should > > > > > > > >> > > > > >> allow to specify both affinity and anti affinity > > > using > > > > > > > >> > expressions. > > > > > > > >> > > > For > > > > > > > >> > > > > >> example LOCALITY_CONTAINER = "{app1.op1 or > > app1.op2} > > > > and > > > > > > > >> > > {!app2.op1}". > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> > > > > > > > >> > > > > >> On 1/25/16 09:39, Thomas Weise wrote: > > > > > > > >> > > > > >> > > > > > > > >> > > > > >>> Agreed. Originally I thought you wanted both > > > operators > > > > > on > > > > > > > the > > > > > > > >> > same > > > > > > > >> > > > > host. > > > > > > > >> > > > > >>> To > > > > > > > >> > > > > >>> pin them to the same container the same idea can > > be > > > > > > applied, > > > > > > > >> > > through > > > > > > > >> > > > a > > > > > > > >> > > > > >>> "LOCALITY_CONTAINER" attribute. > > > > > > > >> > > > > >>> > > > > > > > >> > > > > >>> > > > > > > > >> > > > > >>> On Mon, Jan 25, 2016 at 9:18 AM, Vlad Rozov < > > > > > > > >> > > [email protected] > > > > > > > >> > > > > > > > > > > > >> > > > > >>> wrote: > > > > > > > >> > > > > >>> > > > > > > > >> > > > > >>> Works as a workaround and requires two extra > ports > > > > that > > > > > > > >> operators > > > > > > > >> > > > > >>>> designer > > > > > > > >> > > > > >>>> may not necessarily provide out of the box. We > > may > > > > > apply > > > > > > > the > > > > > > > >> > same > > > > > > > >> > > > hack > > > > > > > >> > > > > >>>> to > > > > > > > >> > > > > >>>> anti-affinity and request that any two > operators > > > with > > > > > > > >> > > anti-affinity > > > > > > > >> > > > > rule > > > > > > > >> > > > > >>>> are connected by non functional stream, but it > > does > > > > not > > > > > > > look > > > > > > > >> > like > > > > > > > >> > > a > > > > > > > >> > > > > good > > > > > > > >> > > > > >>>> option :). > > > > > > > >> > > > > >>>> > > > > > > > >> > > > > >>>> > > > > > > > >> > > > > >>>> > > > > > > > >> > > > > >>>> On 1/25/16 08:52, Thomas Weise wrote: > > > > > > > >> > > > > >>>> > > > > > > > >> > > > > >>>> Sandeep suggested to connect them with a > stream. > > > > > > > >> > > > > >>>>> > > > > > > > >> > > > > >>>>> On Mon, Jan 25, 2016 at 8:49 AM, Vlad Rozov < > > > > > > > >> > > > [email protected] > > > > > > > >> > > > > > > > > > > > > >> > > > > >>>>> wrote: > > > > > > > >> > > > > >>>>> > > > > > > > >> > > > > >>>>> Sorry, possibly I miss something. X and Z are > > not > > > > > > directly > > > > > > > >> > > > connected > > > > > > > >> > > > > >>>>> by a > > > > > > > >> > > > > >>>>> > > > > > > > >> > > > > >>>>>> stream, how can I apply THREAD_LOCAL? > > > > > > > >> > > > > >>>>>> > > > > > > > >> > > > > >>>>>> > > > > > > > >> > > > > >>>>>> On 1/25/16 08:44, Thomas Weise wrote: > > > > > > > >> > > > > >>>>>> > > > > > > > >> > > > > >>>>>> Why does THREAD_LOCAL not work? > > > > > > > >> > > > > >>>>>> > > > > > > > >> > > > > >>>>>>> On Mon, Jan 25, 2016 at 7:43 AM, Vlad Rozov > < > > > > > > > >> > > > > [email protected] > > > > > > > >> > > > > >>>>>>> > > > > > > > > >> > > > > >>>>>>> wrote: > > > > > > > >> > > > > >>>>>>> > > > > > > > >> > > > > >>>>>>> Neither LOCALITY_HOST or THREAD_LOCAL > provides > > > > > > required > > > > > > > >> > > > > >>>>>>> functionality. > > > > > > > >> > > > > >>>>>>> The > > > > > > > >> > > > > >>>>>>> > > > > > > > >> > > > > >>>>>>> goal is to deploy X and Z to the same > > > > > container/JVM. X > > > > > > > >> may be > > > > > > > >> > > > > >>>>>>>> deployed > > > > > > > >> > > > > >>>>>>>> on > > > > > > > >> > > > > >>>>>>>> any node. Z needs to be deployed on the > same > > > > > > container > > > > > > > >> as X, > > > > > > > >> > > so > > > > > > > >> > > > X > > > > > > > >> > > > > >>>>>>>> does > > > > > > > >> > > > > >>>>>>>> not > > > > > > > >> > > > > >>>>>>>> have any host affinity, and Z has affinity > to > > > X, > > > > > not > > > > > > to > > > > > > > >> any > > > > > > > >> > > > > specific > > > > > > > >> > > > > >>>>>>>> host. > > > > > > > >> > > > > >>>>>>>> If database client caches writes, reading > > > inside > > > > > the > > > > > > > same > > > > > > > >> > JVM > > > > > > > >> > > > will > > > > > > > >> > > > > >>>>>>>> be > > > > > > > >> > > > > >>>>>>>> in > > > > > > > >> > > > > >>>>>>>> process lookup, while deploying Z to any > > other > > > > > > host/JVM > > > > > > > >> > leads > > > > > > > >> > > to > > > > > > > >> > > > > >>>>>>>> inter > > > > > > > >> > > > > >>>>>>>> process or inter host lookup. > > > > > > > >> > > > > >>>>>>>> > > > > > > > >> > > > > >>>>>>>> > > > > > > > >> > > > > >>>>>>>> On 1/24/16 21:03, Thomas Weise wrote: > > > > > > > >> > > > > >>>>>>>> > > > > > > > >> > > > > >>>>>>>> There are the attributes LOCALITY_HOST and > > > > > > > LOCALITY_RACK > > > > > > > >> for > > > > > > > >> > > an > > > > > > > >> > > > > >>>>>>>> operator > > > > > > > >> > > > > >>>>>>>> > > > > > > > >> > > > > >>>>>>>> to > > > > > > > >> > > > > >>>>>>>>> achieve precisely what you are describing. > > The > > > > > value > > > > > > > >> would > > > > > > > >> > be > > > > > > > >> > > > an > > > > > > > >> > > > > >>>>>>>>> alias > > > > > > > >> > > > > >>>>>>>>> that > > > > > > > >> > > > > >>>>>>>>> can be mapped to physical hosts for > > > allocation. > > > > > The > > > > > > > same > > > > > > > >> > > could > > > > > > > >> > > > be > > > > > > > >> > > > > >>>>>>>>> used > > > > > > > >> > > > > >>>>>>>>> of > > > > > > > >> > > > > >>>>>>>>> anti-affinity, by assigning different > > values. > > > > > > > >> > > > > >>>>>>>>> > > > > > > > >> > > > > >>>>>>>>> Stream locality is a special case of > > > expressing > > > > > > > affinity > > > > > > > >> > for > > > > > > > >> > > 2 > > > > > > > >> > > > > >>>>>>>>> operators. > > > > > > > >> > > > > >>>>>>>>> Note that we also need a way to extend > this > > to > > > > > > > >> partitions > > > > > > > >> > of > > > > > > > >> > > > the > > > > > > > >> > > > > >>>>>>>>> same > > > > > > > >> > > > > >>>>>>>>> operator. > > > > > > > >> > > > > >>>>>>>>> > > > > > > > >> > > > > >>>>>>>>> > > > > > > > >> > > > > >>>>>>>>> On Sun, Jan 24, 2016 at 8:52 PM, Vlad > Rozov > > < > > > > > > > >> > > > > >>>>>>>>> [email protected]> > > > > > > > >> > > > > >>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>> > > > > > > > >> > > > > >>>>>>>>> It may be quite extreme use case, but > > suppose > > > > that > > > > > > > >> > operator X > > > > > > > >> > > > > >>>>>>>>> writes > > > > > > > >> > > > > >>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>> a > > > > > > > >> > > > > >>>>>>>>> > > > > > > > >> > > > > >>>>>>>>> distributed in-memory database and > operator > > Z > > > > > reads > > > > > > > from > > > > > > > >> > it. > > > > > > > >> > > Z > > > > > > > >> > > > is > > > > > > > >> > > > > >>>>>>>>> not > > > > > > > >> > > > > >>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>> directly connected to X. In such cases it > > may > > > > be > > > > > > > >> necessary > > > > > > > >> > > to > > > > > > > >> > > > > let > > > > > > > >> > > > > >>>>>>>>>> an > > > > > > > >> > > > > >>>>>>>>>> application request container affinity > > > > (deploying > > > > > > to > > > > > > > >> the > > > > > > > >> > > same > > > > > > > >> > > > > JVM) > > > > > > > >> > > > > >>>>>>>>>> for > > > > > > > >> > > > > >>>>>>>>>> operators X and Z as writes may be cached > > and > > > > > reads > > > > > > > >> from > > > > > > > >> > the > > > > > > > >> > > > > same > > > > > > > >> > > > > >>>>>>>>>> JVM > > > > > > > >> > > > > >>>>>>>>>> will > > > > > > > >> > > > > >>>>>>>>>> be potentially faster. > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>> Technically the same may be applied to > > thread > > > > > local > > > > > > > on > > > > > > > >> > NUMA > > > > > > > >> > > > > >>>>>>>>>> boxes, as > > > > > > > >> > > > > >>>>>>>>>> NUMA > > > > > > > >> > > > > >>>>>>>>>> aware deployment of containers is not > > > > supported. > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>> Vlad > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>> On 1/23/16 18:01, Yogi Devendra wrote: > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>> @Isha > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>> In my opinion: > > > > > > > >> > > > > >>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>> THREAD_LOCAL, CONTAINER_LOCAL on stream > > is a > > > > > > special > > > > > > > >> case > > > > > > > >> > > of > > > > > > > >> > > > > >>>>>>>>>>> generic > > > > > > > >> > > > > >>>>>>>>>>> rules > > > > > > > >> > > > > >>>>>>>>>>> for Operator X and Operator Y. > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>> We can say that, THREAD_LOCAL, > > > CONTAINER_LOCAL > > > > > > would > > > > > > > >> be > > > > > > > >> > > > > >>>>>>>>>>> applicable > > > > > > > >> > > > > >>>>>>>>>>> only > > > > > > > >> > > > > >>>>>>>>>>> if > > > > > > > >> > > > > >>>>>>>>>>> operator X and Y are connected by > stream. > > > But, > > > > > way > > > > > > > to > > > > > > > >> > > express > > > > > > > >> > > > > >>>>>>>>>>> this > > > > > > > >> > > > > >>>>>>>>>>> should > > > > > > > >> > > > > >>>>>>>>>>> be similar to other rules for affinity. > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>> ~ Yogi > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>> On 24 January 2016 at 03:49, Isha > > Arkatkar < > > > > > > > >> > > > > [email protected] > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > > >> > > > > >>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>> Hey Chinmay, > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>> I certainly agree on common set > of > > > > rules > > > > > > for > > > > > > > >> > > > > configuring > > > > > > > >> > > > > >>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>> affinity! > > > > > > > >> > > > > >>>>>>>>>>>> Well > > > > > > > >> > > > > >>>>>>>>>>>> put by a concrete example. :) > > > > > > > >> > > > > >>>>>>>>>>>> Only thing I would like to > point > > > is: > > > > > > > >> affinity of > > > > > > > >> > > > > >>>>>>>>>>>> operators > > > > > > > >> > > > > >>>>>>>>>>>> should > > > > > > > >> > > > > >>>>>>>>>>>> not > > > > > > > >> > > > > >>>>>>>>>>>> cover thread and container locality. > > Since > > > > this > > > > > > is > > > > > > > >> only > > > > > > > >> > > > > >>>>>>>>>>>> applicable > > > > > > > >> > > > > >>>>>>>>>>>> when > > > > > > > >> > > > > >>>>>>>>>>>> operators are connected by stream. So, > it > > > > makes > > > > > > > >> sense to > > > > > > > >> > > > have > > > > > > > >> > > > > >>>>>>>>>>>> it on > > > > > > > >> > > > > >>>>>>>>>>>> Stream > > > > > > > >> > > > > >>>>>>>>>>>> rather than in common configuration. > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> And yes, DAG.validate should > only > > > > check > > > > > > for > > > > > > > >> > > REQUIRED > > > > > > > >> > > > or > > > > > > > >> > > > > >>>>>>>>>>>> STRICT > > > > > > > >> > > > > >>>>>>>>>>>> policy. We > > > > > > > >> > > > > >>>>>>>>>>>> can agree on one of the terminologies > > > > > > > STRICT/RELAXED > > > > > > > >> or > > > > > > > >> > > > > >>>>>>>>>>>> REQUIRED/PREFERRED. > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> Thanks! > > > > > > > >> > > > > >>>>>>>>>>>> Isha > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> On Fri, Jan 22, 2016 at 8:38 PM, > Chinmay > > > > > > Kolhatkar > > > > > > > < > > > > > > > >> > > > > >>>>>>>>>>>> [email protected]> > > > > > > > >> > > > > >>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> Hi Isha, Bhupesh, > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> When I suggested singe affinity rule, I > > was > > > > > > mainly > > > > > > > >> > talking > > > > > > > >> > > > > about > > > > > > > >> > > > > >>>>>>>>>>>> "how > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> to" > > > > > > > >> > > > > >>>>>>>>>>>>> of configuration and not of > > > implementation. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> I see locality is in a way suggesting > an > > > > > > affinity > > > > > > > of > > > > > > > >> > > > > operators. > > > > > > > >> > > > > >>>>>>>>>>>>> They're > > > > > > > >> > > > > >>>>>>>>>>>>> close terminologies. > > > > > > > >> > > > > >>>>>>>>>>>>> By configuring a locality on stream, > > we're > > > > > also, > > > > > > > in > > > > > > > >> a > > > > > > > >> > > way, > > > > > > > >> > > > > >>>>>>>>>>>>> defining > > > > > > > >> > > > > >>>>>>>>>>>>> affinity of operators. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> Until now, only locality was there and > > > hence > > > > > was > > > > > > > >> > straight > > > > > > > >> > > > > >>>>>>>>>>>>> forward > > > > > > > >> > > > > >>>>>>>>>>>>> in > > > > > > > >> > > > > >>>>>>>>>>>>> configuration for user. > > > > > > > >> > > > > >>>>>>>>>>>>> Tomorrow, when anti-affinity > > configuration > > > > > comes > > > > > > > up, > > > > > > > >> > one > > > > > > > >> > > > > might > > > > > > > >> > > > > >>>>>>>>>>>>> get > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> confused > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> on how to best use both locality & > > > > > > anti-affinity. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> Hence suggested to make both > > > > > (locality/affinity & > > > > > > > >> > > > > >>>>>>>>>>>> anti-affinity) as > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> a > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> part > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> of single configuration. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> Suggestion is to have a more commonly > > > adopted > > > > > > > >> > > configuration > > > > > > > >> > > > > >>>>>>>>>>>> which > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> admins > > > > > > > >> > > > > >>>>>>>>>>>>> and developer's are familiar with. > > > > > > > >> > > > > >>>>>>>>>>>>> Again referring to vShere Hypervisor's > > > > > affinity > > > > > > > >> rules. > > > > > > > >> > I > > > > > > > >> > > > > think > > > > > > > >> > > > > >>>>>>>>>>>>> they > > > > > > > >> > > > > >>>>>>>>>>>>> have > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> a > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> single configuration which does both. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> Having said that, here is a quick > > > suggestion > > > > on > > > > > > how > > > > > > > >> both > > > > > > > >> > > can > > > > > > > >> > > > > be > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> achieved > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> in > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> a single configuration: > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> CATEGORY TYPE POLICY > > > > > > > >> ENTITIES > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> Affinity THREAD > > REQUIRED > > > > > > > >> O1, O2 > > > > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > > > > >> > > > > >>>>>>>>>>>>> Operator1 & Operator2 should be thread > > > local > > > > > > > >> > > > > >>>>>>>>>>>>> Affinity NODE > > > PREFERRED > > > > > > > >> O3, O4 > > > > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> O3 & > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> O4 are preferred to be in node local > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> AntiAffinity NODE > REQUIRED > > > > > > > O1, > > > > > > > >> O4 > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> O1 & > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> O4 should not be on same node. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> AntiAffinity RACK > > PREFERRED > > > > > > O2, > > > > > > > O4 > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> //Meaning > > > > > > > >> > > > > >>>>>>>>>>>>> O2 > > > > > > > >> > > > > >>>>>>>>>>>>> & > > > > > > > >> > > > > >>>>>>>>>>>>> O4 are preferred not to be on same > rack. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> Linux setting affinity of CPUs for > > threads > > > > is > > > > > > > >> another > > > > > > > >> > way > > > > > > > >> > > > of > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> configuration > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> we can take a look at. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> Learning from these commonly adopted > > > > > > configuration > > > > > > > >> > > pattern, > > > > > > > >> > > > we > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> should > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> come > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> up with best configuration suitable > for > > > > > > > distributed > > > > > > > >> > > > > >>>>>>>>>>>>> environment. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> Idea here is to not have our own > > > > configuration > > > > > > and > > > > > > > >> give > > > > > > > >> > > > > >>>>>>>>>>>> something > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> new > > > > > > > >> > > > > >>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>> the users. Otherwise such an important > > > > concept > > > > > > > might > > > > > > > >> > > > quickly > > > > > > > >> > > > > >>>>>>>>>>>>> get > > > > > > > >> > > > > >>>>>>>>>>>>> lost. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> Regarding the DAG.validate I think we > > > would > > > > > need > > > > > > > to > > > > > > > >> add > > > > > > > >> > > > some > > > > > > > >> > > > > >>>>>>>>>>>>> new > > > > > > > >> > > > > >>>>>>>>>>>>> stuff > > > > > > > >> > > > > >>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>> take care of anti-affinity. > > > > > > > >> > > > > >>>>>>>>>>>>> Plus, anti-affinity/affinity should be > > > > > validated > > > > > > > at > > > > > > > >> > > > > >>>>>>>>>>>>> DAG.validate > > > > > > > >> > > > > >>>>>>>>>>>>> only > > > > > > > >> > > > > >>>>>>>>>>>>> for > > > > > > > >> > > > > >>>>>>>>>>>>> the ones which are required. > > > > > > > >> > > > > >>>>>>>>>>>>> For preferred policies, validation in > > > > logical > > > > > > plan > > > > > > > >> > might > > > > > > > >> > > > be a > > > > > > > >> > > > > >>>>>>>>>>>>> early > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> check. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>> Chinmay. > > > > > > > >> > > > > >>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> On Sat, Jan 23, 2016 at 3:15 AM, Isha > > > > > Arkatkar < > > > > > > > >> > > > > >>>>>>>>>>>>> [email protected]> > > > > > > > >> > > > > >>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> Hi, > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> Thanks for inputs! I like the > > idea > > > of > > > > > > > having > > > > > > > >> > > single > > > > > > > >> > > > > set > > > > > > > >> > > > > >>>>>>>>>>>>> of > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> (anti?)affinity rules at dag context > > > level. > > > > > > > >> > > > > >>>>>>>>>>>>>> There could still be conflicts > > > based > > > > > on > > > > > > > >> > Locality > > > > > > > >> > > > for > > > > > > > >> > > > > >>>>>>>>>>>>>> Streams > > > > > > > >> > > > > >>>>>>>>>>>>>> or > > > > > > > >> > > > > >>>>>>>>>>>>>> Node > > > > > > > >> > > > > >>>>>>>>>>>>>> locality attribute set for operators. > > But > > > > as > > > > > > > >> Sandeep > > > > > > > >> > > > > >>>>>>>>>>>>>> suggested, I > > > > > > > >> > > > > >>>>>>>>>>>>>> also > > > > > > > >> > > > > >>>>>>>>>>>>>> think Dag.validate should fail in > case > > of > > > > > > > >> > contradicting > > > > > > > >> > > > > >>>>>>>>>>>>>> constraints. > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> We currently do not have > > 'affinity' > > > > > among > > > > > > > non > > > > > > > >> > > stream > > > > > > > >> > > > > >>>>>>>>>>>>>> operators, as > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> Bhupesh > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> pointed out. It is somewhat > achievable > > by > > > > > > > >> requesting > > > > > > > >> > > node > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> locality > > > > > > > >> > > > > >>>>>>>>>>>>> for > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> operators (Assuming node requests work > > as > > > > > > > expected). > > > > > > > >> > But > > > > > > > >> > > > > >>>>>>>>>>>>> should we > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> consider > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> adding affinity specifications > support > > as > > > > > well > > > > > > > >> along > > > > > > > >> > > with > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> anti-affinity? > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> Regarding specification of > > > attributes > > > > > > from > > > > > > > >> > > > > dt-site.xml. > > > > > > > >> > > > > >>>>>>>>>>>>>> We > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> can > > > > > > > >> > > > > >>>>>>>>>>>>> go > > > > > > > >> > > > > >>>>>>>>>>>>> with > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> Json like string or even xml > > > representation > > > > > for > > > > > > > >> complex > > > > > > > >> > > > > >>>>>>>>>>>>> objects. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> What > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> is > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> our current behavior for setting Java > > > > object > > > > > > > >> > properties > > > > > > > >> > > > > >>>>>>>>>>>>>> through > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> XML? > > > > > > > >> > > > > >>>>>>>>>>>>> We > > > > > > > >> > > > > >>>>>>>>>>>>> can > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> follow the same for this as well. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> As for precedence or ability to > > > > satisfy > > > > > > > >> > > constraints: > > > > > > > >> > > > > >>>>>>>>>>>>> Right > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> now in > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> normal > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> scenario, if resources are not > > available > > > > for > > > > > > > >> > allocating > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> containers, > > > > > > > >> > > > > >>>>>>>>>>>>> we > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> keep > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> sending to request till all are > > obtained. > > > > > > > >> Likewise, in > > > > > > > >> > > > case > > > > > > > >> > > > > of > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> strict > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> anti-affinity policy, we should keep > the > > > > > > > >> application in > > > > > > > >> > > > > >>>>>>>>>>>>> ACCEPTED > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> state > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> till > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> the anti-affinity constraint is > > > satisfied. > > > > > For > > > > > > > >> relaxed > > > > > > > >> > > > > >>>>>>>>>>>>>> policy, we > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> can > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> decide timeout for relaxing the > > > > anti-affinity > > > > > > > rule. > > > > > > > >> > > Please > > > > > > > >> > > > > note > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> this > > > > > > > >> > > > > >>>>>>>>>>>>>> applies only when we have > > > non-contradicting > > > > > > > rules. > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> Thanks! > > > > > > > >> > > > > >>>>>>>>>>>>>> Isha > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 5:30 AM, > > Bhupesh > > > > > > Chawda < > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> [email protected] > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> I agree on having a single set of > rules > > > for > > > > > the > > > > > > > >> > affinity > > > > > > > >> > > as > > > > > > > >> > > > > >>>>>>>>>>>>> well > > > > > > > >> > > > > >>>>>>>>>>>>> as > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> anti-affinity of operators / > partitions > > on > > > > > > > >> containers. > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> However, I noted the following > points: > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> 1. AFAIK, we do not support > > > > affinity > > > > > > > >> > (locality) > > > > > > > >> > > > in > > > > > > > >> > > > > a > > > > > > > >> > > > > >>>>>>>>>>>>>>> general > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> sense. > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> The > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> affinity is only for a > stream, > > > not > > > > > for > > > > > > > >> *any* > > > > > > > >> > two > > > > > > > >> > > > > >>>>>>>>>>>>> operators. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> So, > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> we > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> should also look at the > > general > > > > case > > > > > > and > > > > > > > >> see > > > > > > > >> > > how > > > > > > > >> > > > it > > > > > > > >> > > > > >>>>>>>>>>>>>>> can > > > > > > > >> > > > > >>>>>>>>>>>>>>> be > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> supported, > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> if > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> there are valid use cases. > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> 2. Coming to anti-affinity, > we > > > > cannot > > > > > > > type > > > > > > > >> > cast > > > > > > > >> > > it > > > > > > > >> > > > > as > > > > > > > >> > > > > >>>>>>>>>>>>>> a > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> type of > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> affinity > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> rule. Saying "two operators > > must > > > > be > > > > > on > > > > > > > the > > > > > > > >> > same > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> container" is > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> very > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> different from saying "these > > two > > > > > > > operators > > > > > > > >> > must > > > > > > > >> > > > not > > > > > > > >> > > > > >>>>>>>>>>>>>>> be on > > > > > > > >> > > > > >>>>>>>>>>>>>>> the > > > > > > > >> > > > > >>>>>>>>>>>>>>> same > > > > > > > >> > > > > >>>>>>>>>>>>>>> container". In this case, > the > > > > second > > > > > > one > > > > > > > >> is a > > > > > > > >> > > > much > > > > > > > >> > > > > >>>>>>>>>>>>>>> more > > > > > > > >> > > > > >>>>>>>>>>>>>>> relaxed > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> rule > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> as > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> compared to the first one. > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> 3. Once we have this generic > set > > > of > > > > > > rules, > > > > > > > >> > there > > > > > > > >> > > > must > > > > > > > >> > > > > >>>>>>>>>>>>>> be a > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> "satisfiability" test run > > before > > > > > > > >> requesting > > > > > > > >> > > YARN > > > > > > > >> > > > > for > > > > > > > >> > > > > >>>>>>>>>>>>>>> containers. > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> If > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> the > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> request is not satisfiable, > > then > > > > > there > > > > > > is > > > > > > > >> no > > > > > > > >> > > point > > > > > > > >> > > > > >>>>>>>>>>>>> asking > > > > > > > >> > > > > >>>>>>>>>>>>> YARN > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> allocate > > > > > > > >> > > > > >>>>>>>>>>>>>>> containers in this manner. > In > > > case > > > > > it > > > > > > is > > > > > > > >> not > > > > > > > >> > > > > >>>>>>>>>>>>>>> satisfiable, we > > > > > > > >> > > > > >>>>>>>>>>>>>>> must > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> also > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> have > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> a default order in which the > > > rules > > > > > can > > > > > > be > > > > > > > >> > > > "relaxed" > > > > > > > >> > > > > >>>>>>>>>>>>>> and > > > > > > > >> > > > > >>>>>>>>>>>>>> the > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> request > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> be > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> made > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> satisfiable. For example, some > > > very > > > > > > strict > > > > > > > >> > rules > > > > > > > >> > > > may > > > > > > > >> > > > > be > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> ignored, > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> or > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> made > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> less constraining ( for > example > > > "on > > > > > the > > > > > > > >> same > > > > > > > >> > > > > >>>>>>>>>>>>> container" => > > > > > > > >> > > > > >>>>>>>>>>>>> "on > > > > > > > >> > > > > >>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>> the > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> same > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> node"). > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> Thanks. > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> -Bhupesh > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 2:54 PM, > > > Aniruddha > > > > > > > >> Thombare < > > > > > > > >> > > > > >>>>>>>>>>>>>>> [email protected]> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> +1 On Chinmay's suggestion about > > having > > > > > single > > > > > > > >> set of > > > > > > > >> > > > > >>>>>>>>>>>>>>> affinity > > > > > > > >> > > > > >>>>>>>>>>>>>>> rules. > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> Aniruddha > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:57 PM, > > > Sandeep > > > > > > > >> Deshmukh < > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> [email protected] > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> Between 2 operators, if one > configures > > > > > > > >> > thread/container > > > > > > > >> > > > > local > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> and > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> anti-affinity as well, which one > will > > > > take > > > > > > > >> affect? > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> The DAG validation step should > error > > > out > > > > > in > > > > > > > this > > > > > > > >> > > case. > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> +1 on suggestion by Chinmay to name > > it > > > > > > > "Affinity > > > > > > > >> > > Rules" > > > > > > > >> > > > > >>>>>>>>>>>>>>> than > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> anti-affinity. We are just extending > > our > > > > > > > container > > > > > > > >> > > > > allocation > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> scheme > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> support containers not to be > > allocated > > > > > > > together. > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> Regards, > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> Sandeep > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 1:43 PM, > > > Chinmay > > > > > > > >> Kolhatkar > > > > > > > >> > < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> Hi Isha, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> Couple of points: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> 1. About the interface to > > configuring > > > > > > > >> > anti-affinity, > > > > > > > >> > > as > > > > > > > >> > > > > per > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> suggestion > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> above there are 2 different way > to > > > > > > configure > > > > > > > >> > > locality > > > > > > > >> > > > > and > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> anti-affinity: > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> i.e. dag.setAttribute - for > > > anti-affinity > > > > > & > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > dag.addStream(...).setLocality > > > for > > > > > > > >> locality. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> Between 2 operators, if one > > configures > > > > > > > >> > > thread/container > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> local > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> and > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity as well, which one > > will > > > > > take > > > > > > > >> affect? > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> 2. Consider there could be such > > > > confusion > > > > > as > > > > > > > >> above, > > > > > > > >> > > > would > > > > > > > >> > > > > >>>>>>>>>>>>>>>> it > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> make > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> sense > > > > > > > >> > > > > >>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> have a single API which takes care > of > > > both > > > > > > > >> > > anti-affinity > > > > > > > >> > > > > and > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> locality. > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> This > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> way, one is configurable. > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> 3. This point is coming from how VM > > > > > affinity > > > > > > is > > > > > > > >> > > > configured > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> in > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> vSphere. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> The VMs are configured affinity > are > > > > > called > > > > > > as > > > > > > > >> > > > "affinity > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> rules" > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> and > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> not > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> "anti-affinity rules". Ultimately > > idea > > > > is > > > > > to > > > > > > > >> > allocate > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> processing > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> nodes. > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> Via "VM-VM affinity rules", > > > anti-affinity > > > > is > > > > > > > also > > > > > > > >> > > > > configured. > > > > > > > >> > > > > >>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>> But > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> there > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> is > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> a single set of rule definition for > > both > > > > > > > affinity > > > > > > > >> > > > (similar > > > > > > > >> > > > > to > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> locality > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> in > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> our case) and anti-affinity. > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> Would it be a better approach for > > > > > configuring > > > > > > > >> > locality > > > > > > > >> > > > > rules > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> and > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> anti-affinity rules in a single > > rule > > > > and > > > > > > call > > > > > > > >> it > > > > > > > >> > > > > "affinity > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> rule". > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > >>>>>>>>>>>>>>> Chinmay. > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>> On Fri, Jan 22, 2016 at 12:24 PM, > Yogi > > > > > > Devendra > > > > > > > < > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected] > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> @Isha > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> I understand that anti-affinity > > across > > > > > > > >> application > > > > > > > >> > is > > > > > > > >> > > > not > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> straight-forward. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> It would be OK even if we do not > > > have > > > > it > > > > > > in > > > > > > > >> > > iteration > > > > > > > >> > > > > 1. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> But, for attributes syntax; I > still > > > > think > > > > > > > that > > > > > > > >> > Java > > > > > > > >> > > > > object > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> should > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> be > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> avoided as they will be hard to > > > > configure > > > > > > > from > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> dt-site.xml or > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> other > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> config > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> files. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> Other suggestion for this could be > > JSON > > > > > > > >> > representation > > > > > > > >> > > > of > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> String > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> array: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> ["O2", "O3"]. (If operator names > > has > > > > > some > > > > > > > >> special > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> characters > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> like > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> " > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> or [ > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> or , those will be escaped in the > > JSON > > > > > > > >> > > representation.) > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> Not sure if others agree on this; > but > > > > > > attribute > > > > > > > >> > syntax > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> should > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> be > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> finalized > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> in iteration 1 to avoid backward > > > > > > > compatibility > > > > > > > >> > > issues > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> later. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> ~ Yogi > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> On 22 January 2016 at 00:43, Thomas > > > > Weise < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected]> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Node based requests is the best > > > > approach > > > > > - > > > > > > if > > > > > > > >> it > > > > > > > >> > > works > > > > > > > >> > > > > :-) > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> Blacklisting will require to > allocate > > > the > > > > > > > >> containers > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> sequentially. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> It > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> will > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> work, but slow down application > > > startup, > > > > > > > >> especially > > > > > > > >> > > for > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> larger > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> topologies. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> On Thu, Jan 21, 2016 at 10:42 > AM, > > > Isha > > > > > > > >> Arkatkar < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> [email protected]> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Hi, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Should we consider the > node > > > > based > > > > > > > >> requests > > > > > > > >> > if > > > > > > > >> > > > it > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> works > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> with > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Capacity > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> Scheduler or avoid 2b approach > > > > > > altogether? > > > > > > > I > > > > > > > >> > > checked > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> that > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> node > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> requests > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> do > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> not work with fair scheduler on > CDH > > > > > > cluster. > > > > > > > >> Yarn > > > > > > > >> > > does > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> not > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> return > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> any > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> container if hostname is given in > > the > > > > > > > container > > > > > > > >> > > > request. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> I > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> am > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> trying > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> setup a small virtual hortonworks > > > cluster > > > > > to > > > > > > > >> check > > > > > > > >> > the > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> this > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> behavior > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> on > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> that. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> YARN-2027 < > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > https://issues.apache.org/jira/browse/YARN-2027 > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> mentioned > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> that > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> container requests are not > honored > > in > > > > > > > capacity > > > > > > > >> > > > scheduler > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> too. > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> But I > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> am > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> not > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> sure if it is because of distro > > > > dependent > > > > > > > issue. > > > > > > > >> > > Please > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> share > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> insights. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> @Vlad, Adding support for > regular > > > > > > > expression > > > > > > > >> > > sounds > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> good. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> We > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> could > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> translate to list of operator > names > > > > > > > internally > > > > > > > >> > based > > > > > > > >> > > > on > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> regex. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> @Yogi, I went with a list of > > strings > > > > for > > > > > > > >> attribute > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> because > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> "O2, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> O3" > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> could > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> be a valid single operator name too > > :) > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> I am not sure of ways to implement > > > > > > > anti-affinity > > > > > > > >> > > across > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> application. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Though > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> something to consider for later > > > > > > iteration. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> Isha > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:59 > PM, > > > > > Thomas > > > > > > > >> Weise < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> https://issues.apache.org/jira/browse/SLIDER-82 > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:56 PM, > > > > Thomas > > > > > > > Weise > > > > > > > >> < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> [email protected]> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> The point was that containers > > are > > > > > taken > > > > > > > away > > > > > > > >> > from > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> other > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> apps > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> that > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> may > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> have > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> to discard work etc. It's not good > > > style > > > > > to > > > > > > > >> claim > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> resources > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> and > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> not > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> use > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> them eventually :-) > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> For this feature it is necessary > > to > > > > look > > > > > > at > > > > > > > >> the > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> scheduler > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> capabilities/semantics and > > > > > > limitations. > > > > > > > >> For > > > > > > > >> > > > > example, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> don't > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> bet > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> exclusively > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> on node requests if the goal is > for > > > it > > > > to > > > > > > > work > > > > > > > >> > with > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> FairScheduler. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Also look at Slider, which just > > > > recently > > > > > > > added > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> support > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> for > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> anti-affinity > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> (using node requests). When you > > run > > > > it > > > > > on > > > > > > > the > > > > > > > >> > CDH > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> cluster, > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> it > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> probably > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> won't work... > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 3:19 PM, > > > Pramod > > > > > > > >> Immaneni < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> [email protected] > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Once released won't the > > > containers > > > > be > > > > > > > >> > available > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> again > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> in > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> the > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> pool. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> This > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> would only be optional and not > > > mandatory. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> Thanks > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at 2:02 > PM, > > > > Thomas > > > > > > > >> Weise < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected] > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> How about also supporting a > > > minor > > > > > > > >> variation > > > > > > > >> > of > > > > > > > >> > > it > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> as > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> an > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> option > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> where it greedily gets the > > total > > > > > number > > > > > > > of > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> containers > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> and > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> discards > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ones > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> it > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> can't use and repeats the > process > > > for > > > > > the > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> remaining > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> till > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> everything > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> has > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> been allocated. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> This is problematic as with > > resource > > > > > > > preemption > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> these > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> containers > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> will > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> potentially taken away from > other > > > > > > > >> applications > > > > > > > >> > and > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> then > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> thrown > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> away. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Also does it make sense to > > > support > > > > > > > >> > anti-cluster > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> affinity? > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> Thanks > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at > 1:21 > > > PM, > > > > > > Isha > > > > > > > >> > > Arkatkar > > > > > > > >> > > > < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> [email protected]> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> wrote: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> Hi all, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> We want add support > > for > > > > > > > >> > Anti-affinity > > > > > > > >> > > in > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Apex > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> allow > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> applications > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> launch specific physical operators > > on > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> different > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> nodes(APEXCORE-10 > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > https://issues.apache.org/jira/browse/APEXCORE-10 > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ). > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> Want > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> request > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> your > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> suggestions/ideas for the same! > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> The reasons for using > > > > > > > anti-affinity > > > > > > > >> in > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> could > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> be: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> ensure > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> reliability, for performance > > reasons > > > > > (such > > > > > > as > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> application > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> may > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> not > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> want > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> 2 > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> i/o intensive operators to > land > > > on > > > > > the > > > > > > > same > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> node > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> improve > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> performance) > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> or > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> for some application specific > > > > > > constraints(for > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> example, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> 2 > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> partitions > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> cannot > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> be run on the same node since > > > they > > > > > use > > > > > > > same > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> port > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> number). > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> This > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> is > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> general rationale for adding > > > > Anti-affinity > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> support. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Since, Yarn does not > support > > > > > > > >> anti-affinity > > > > > > > >> > yet > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> (YARN-1042 > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> < > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > https://issues.apache.org/jira/browse/YARN-1042 > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> ), > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> we > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> need > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> to > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> implement > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> the logic in AM. Wanted to get > > your > > > > > views > > > > > > on > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> following > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> aspects > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> for > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> this > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> implementation: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> *1. How to specify > anti-affinity > > > for > > > > > > > physical > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> operators/partitions > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> in > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> application:* > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> One way for this > is > > to > > > > > have > > > > > > an > > > > > > > >> > > > attribute > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> for > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> setting > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> at > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> the logical operator context. > > And > > > > an > > > > > > > >> operator > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> can > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> set > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> this > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> attribute > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> with > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> list of operator names which > should > > > not > > > > > be > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> collocated. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> Consider dag with 3 > > > > > > operators: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> TestOperator > o1 = > > > > > > > >> > > > > dag.addOperator("O1", > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> TestOperator o2 = > > > > > > > >> > dag.addOperator("O2", > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> TestOperator o3 = > > > > > > > >> dag.addOperator("O3", > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> new > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> TestOperator()); > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> To set anti-affinity for O1 > > > > > operator: > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > dag.setAttribute(o1, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > OperatorContext.ANTI_AFFINITY, > > > > > > > >> > > > > >>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> new > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > ArrayList<String>(Arrays.asList("O2", > > > > > > > >> > > "O3"))); > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> This would mean > O1 > > > > should > > > > > > not > > > > > > > >> be > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> allocated > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> on > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> nodes > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> containing > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> operators O2 and O3. This > > applies > > > > to > > > > > > all > > > > > > > >> > > > > >>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>> allocated > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> partitions > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>> of > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> O1, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> O2, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>> O3. > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>> Also, if same > operator > > > > name > > > > > is > > > > > > > >> part > > > > > > > >> > of > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> anti-affinity > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> list, > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> > > > > > > > >> > > > > >> > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
