Why does THREAD_LOCAL not work? On Mon, Jan 25, 2016 at 7:43 AM, Vlad Rozov <[email protected]> wrote:
> Neither LOCALITY_HOST or THREAD_LOCAL provides required functionality. The > goal is to deploy X and Z to the same container/JVM. X may be deployed on > any node. Z needs to be deployed on the same container as X, so X does not > have any host affinity, and Z has affinity to X, not to any specific host. > If database client caches writes, reading inside the same JVM will be in > process lookup, while deploying Z to any other host/JVM leads to inter > process or inter host lookup. > > > On 1/24/16 21:03, Thomas Weise wrote: > >> There are the attributes LOCALITY_HOST and LOCALITY_RACK for an operator >> to >> achieve precisely what you are describing. The value would be an alias >> that >> can be mapped to physical hosts for allocation. The same could be used of >> anti-affinity, by assigning different values. >> >> Stream locality is a special case of expressing affinity for 2 operators. >> Note that we also need a way to extend this to partitions of the same >> operator. >> >> >> On Sun, Jan 24, 2016 at 8:52 PM, Vlad Rozov <[email protected]> >> wrote: >> >> It may be quite extreme use case, but suppose that operator X writes to a >>> distributed in-memory database and operator Z reads from it. Z is not >>> directly connected to X. In such cases it may be necessary to let an >>> application request container affinity (deploying to the same JVM) for >>> operators X and Z as writes may be cached and reads from the same JVM >>> will >>> be potentially faster. >>> >>> Technically the same may be applied to thread local on NUMA boxes, as >>> NUMA >>> aware deployment of containers is not supported. >>> >>> Vlad >>> >>> On 1/23/16 18:01, Yogi Devendra wrote: >>> >>> @Isha >>>> >>>> In my opinion: >>>> THREAD_LOCAL, CONTAINER_LOCAL on stream is a special case of generic >>>> rules >>>> for Operator X and Operator Y. >>>> >>>> We can say that, THREAD_LOCAL, CONTAINER_LOCAL would be applicable only >>>> if >>>> operator X and Y are connected by stream. But, way to express this >>>> should >>>> be similar to other rules for affinity. >>>> >>>> ~ Yogi >>>> >>>> On 24 January 2016 at 03:49, Isha Arkatkar <[email protected]> >>>> wrote: >>>> >>>> Hey Chinmay, >>>> >>>>> I certainly agree on common set of rules for configuring affinity! >>>>> Well >>>>> put by a concrete example. :) >>>>> Only thing I would like to point is: affinity of operators should >>>>> not >>>>> cover thread and container locality. Since this is only applicable when >>>>> operators are connected by stream. So, it makes sense to have it on >>>>> Stream >>>>> rather than in common configuration. >>>>> >>>>> And yes, DAG.validate should only check for REQUIRED or STRICT >>>>> policy. We >>>>> can agree on one of the terminologies STRICT/RELAXED or >>>>> REQUIRED/PREFERRED. >>>>> >>>>> Thanks! >>>>> Isha >>>>> >>>>> >>>>> On Fri, Jan 22, 2016 at 8:38 PM, Chinmay Kolhatkar < >>>>> [email protected]> >>>>> wrote: >>>>> >>>>> Hi Isha, Bhupesh, >>>>> >>>>>> When I suggested singe affinity rule, I was mainly talking about "how >>>>>> to" >>>>>> of configuration and not of implementation. >>>>>> >>>>>> I see locality is in a way suggesting an affinity of operators. >>>>>> They're >>>>>> close terminologies. >>>>>> By configuring a locality on stream, we're also, in a way, defining >>>>>> affinity of operators. >>>>>> >>>>>> Until now, only locality was there and hence was straight forward in >>>>>> configuration for user. >>>>>> Tomorrow, when anti-affinity configuration comes up, one might get >>>>>> >>>>>> confused >>>>> >>>>> on how to best use both locality & anti-affinity. >>>>>> Hence suggested to make both (locality/affinity & anti-affinity) as a >>>>>> >>>>>> part >>>>> >>>>> of single configuration. >>>>>> >>>>>> Suggestion is to have a more commonly adopted configuration which >>>>>> admins >>>>>> and developer's are familiar with. >>>>>> Again referring to vShere Hypervisor's affinity rules. I think they >>>>>> have >>>>>> >>>>>> a >>>>> >>>>> single configuration which does both. >>>>>> >>>>>> Having said that, here is a quick suggestion on how both can be >>>>>> achieved >>>>>> >>>>>> in >>>>> >>>>> a single configuration: >>>>>> >>>>>> CATEGORY TYPE POLICY ENTITIES >>>>>> Affinity THREAD REQUIRED O1, O2 //Meaning >>>>>> Operator1 & Operator2 should be thread local >>>>>> Affinity NODE PREFERRED O3, O4 //Meaning >>>>>> >>>>>> O3 & >>>>> >>>>> O4 are preferred to be in node local >>>>>> AntiAffinity NODE REQUIRED O1, O4 //Meaning >>>>>> >>>>>> O1 & >>>>> >>>>> O4 should not be on same node. >>>>>> AntiAffinity RACK PREFERRED O2, O4 //Meaning >>>>>> O2 >>>>>> & >>>>>> O4 are preferred not to be on same rack. >>>>>> >>>>>> >>>>>> Linux setting affinity of CPUs for threads is another way of >>>>>> >>>>>> configuration >>>>> >>>>> we can take a look at. >>>>>> >>>>>> Learning from these commonly adopted configuration pattern, we should >>>>>> >>>>>> come >>>>> >>>>> up with best configuration suitable for distributed environment. >>>>>> Idea here is to not have our own configuration and give something new >>>>>> to >>>>>> the users. Otherwise such an important concept might quickly get lost. >>>>>> >>>>>> Regarding the DAG.validate I think we would need to add some new stuff >>>>>> to >>>>>> take care of anti-affinity. >>>>>> Plus, anti-affinity/affinity should be validated at DAG.validate only >>>>>> for >>>>>> the ones which are required. >>>>>> For preferred policies, validation in logical plan might be a early >>>>>> >>>>>> check. >>>>> >>>>> Thanks, >>>>>> Chinmay. >>>>>> >>>>>> >>>>>> >>>>>> On Sat, Jan 23, 2016 at 3:15 AM, Isha Arkatkar <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>>> Thanks for inputs! I like the idea of having single set of >>>>>>> (anti?)affinity rules at dag context level. >>>>>>> There could still be conflicts based on Locality for Streams or >>>>>>> Node >>>>>>> locality attribute set for operators. But as Sandeep suggested, I >>>>>>> also >>>>>>> think Dag.validate should fail in case of contradicting constraints. >>>>>>> >>>>>>> We currently do not have 'affinity' among non stream operators, as >>>>>>> >>>>>>> Bhupesh >>>>>> >>>>>> pointed out. It is somewhat achievable by requesting node locality for >>>>>>> operators (Assuming node requests work as expected). But should we >>>>>>> >>>>>>> consider >>>>>> >>>>>> adding affinity specifications support as well along with >>>>>>> >>>>>>> anti-affinity? >>>>>> Regarding specification of attributes from dt-site.xml. We can go >>>>>> with >>>>>> >>>>>>> Json like string or even xml representation for complex objects. What >>>>>>> >>>>>>> is >>>>>> our current behavior for setting Java object properties through XML? >>>>>> We >>>>>> can >>>>>> >>>>>> follow the same for this as well. >>>>>>> >>>>>>> As for precedence or ability to satisfy constraints: Right now in >>>>>>> >>>>>>> normal >>>>>> >>>>>> scenario, if resources are not available for allocating containers, we >>>>>>> >>>>>>> keep >>>>>> >>>>>> sending to request till all are obtained. Likewise, in case of strict >>>>>>> anti-affinity policy, we should keep the application in ACCEPTED >>>>>>> state >>>>>>> >>>>>>> till >>>>>> >>>>>> the anti-affinity constraint is satisfied. For relaxed policy, we can >>>>>>> decide timeout for relaxing the anti-affinity rule. Please note this >>>>>>> applies only when we have non-contradicting rules. >>>>>>> >>>>>>> Thanks! >>>>>>> Isha >>>>>>> >>>>>>> On Fri, Jan 22, 2016 at 5:30 AM, Bhupesh Chawda < >>>>>>> >>>>>>> [email protected] >>>>>> wrote: >>>>>> >>>>>>> I agree on having a single set of rules for the affinity as well as >>>>>>> >>>>>>>> anti-affinity of operators / partitions on containers. >>>>>>>> >>>>>>>> However, I noted the following points: >>>>>>>> >>>>>>>> 1. AFAIK, we do not support affinity (locality) in a general >>>>>>>> >>>>>>>> sense. >>>>>>> >>>>>> The >>>>>> >>>>>>> affinity is only for a stream, not for *any* two operators. So, >>>>>>>> we >>>>>>>> should also look at the general case and see how it can be >>>>>>>> >>>>>>>> supported, >>>>>>> if >>>>>>> >>>>>>> there are valid use cases. >>>>>>>> 2. Coming to anti-affinity, we cannot type cast it as a type of >>>>>>>> >>>>>>>> affinity >>>>>>> >>>>>>> rule. Saying "two operators must be on the same container" is >>>>>>>> very >>>>>>>> different from saying "these two operators must not be on the >>>>>>>> same >>>>>>>> container". In this case, the second one is a much more relaxed >>>>>>>> >>>>>>>> rule >>>>>>> >>>>>> as >>>>>> >>>>>>> compared to the first one. >>>>>>>> 3. Once we have this generic set of rules, there must be a >>>>>>>> "satisfiability" test run before requesting YARN for >>>>>>>> containers. >>>>>>>> >>>>>>>> If >>>>>>> >>>>>> the >>>>>> >>>>>>> request is not satisfiable, then there is no point asking YARN >>>>>>>> to >>>>>>>> allocate >>>>>>>> containers in this manner. In case it is not satisfiable, we >>>>>>>> must >>>>>>>> >>>>>>>> also >>>>>>> have >>>>>>> >>>>>>>> a default order in which the rules can be "relaxed" and the >>>>>>>> >>>>>>>> request >>>>>>> >>>>>> be >>>>>> >>>>>> made >>>>>>> >>>>>>>> satisfiable. For example, some very strict rules may be >>>>>>>> ignored, >>>>>>>> >>>>>>>> or >>>>>>> >>>>>> made >>>>>> >>>>>>> less constraining ( for example "on the same container" => "on >>>>>>>> the >>>>>>>> >>>>>>>> same >>>>>>> >>>>>>> node"). >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> -Bhupesh >>>>>>>> >>>>>>>> On Fri, Jan 22, 2016 at 2:54 PM, Aniruddha Thombare < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>> +1 On Chinmay's suggestion about having single set of affinity >>>>>>>> rules. >>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> >>>>>>>>> Aniruddha >>>>>>>>> >>>>>>>>> On Fri, Jan 22, 2016 at 1:57 PM, Sandeep Deshmukh < >>>>>>>>> >>>>>>>>> [email protected] >>>>>>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Between 2 operators, if one configures thread/container local >>>>>>>>> >>>>>>>>>> and >>>>>>>>>> >>>>>>>>> anti-affinity as well, which one will take affect? >>>>>> >>>>>>> The DAG validation step should error out in this case. >>>>>>>>>> >>>>>>>>>> +1 on suggestion by Chinmay to name it "Affinity Rules" than >>>>>>>>>> anti-affinity. We are just extending our container allocation >>>>>>>>>> >>>>>>>>>> scheme >>>>>>>>> >>>>>>>> to >>>>>>> >>>>>>> support containers not to be allocated together. >>>>>>>> >>>>>>>>> Regards, >>>>>>>>>> Sandeep >>>>>>>>>> >>>>>>>>>> On Fri, Jan 22, 2016 at 1:43 PM, Chinmay Kolhatkar < >>>>>>>>>> [email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Isha, >>>>>>>>>> >>>>>>>>>>> Couple of points: >>>>>>>>>>> 1. About the interface to configuring anti-affinity, as per >>>>>>>>>>> >>>>>>>>>>> suggestion >>>>>>>>>> >>>>>>>>> above there are 2 different way to configure locality and >>>>>>>>> >>>>>>>>>> anti-affinity: >>>>>>>>>> i.e. dag.setAttribute - for anti-affinity & >>>>>>>>>> >>>>>>>>>>> dag.addStream(...).setLocality for locality. >>>>>>>>>>> >>>>>>>>>>> Between 2 operators, if one configures thread/container local >>>>>>>>>>> >>>>>>>>>>> and >>>>>>>>>> >>>>>>>>> anti-affinity as well, which one will take affect? >>>>>> >>>>>>> 2. Consider there could be such confusion as above, would it >>>>>>>>>>> >>>>>>>>>>> make >>>>>>>>>> >>>>>>>>> sense >>>>>> >>>>>>> to >>>>>>>>> >>>>>>>>>> have a single API which takes care of both anti-affinity and >>>>>>>>>>> >>>>>>>>>>> locality. >>>>>>>>>> >>>>>>>>> This >>>>>>>>> >>>>>>>>>> way, one is configurable. >>>>>>>>>>> >>>>>>>>>>> 3. This point is coming from how VM affinity is configured in >>>>>>>>>>> >>>>>>>>>>> vSphere. >>>>>>>>>> >>>>>>>>> The VMs are configured affinity are called as "affinity rules" >>>>>>>>> >>>>>>>>>> and >>>>>>>>>> >>>>>>>>> not >>>>>>> >>>>>>>> "anti-affinity rules". Ultimately idea is to allocate >>>>>>>>> >>>>>>>>>> processing >>>>>>>>>> >>>>>>>>> to >>>>>> >>>>>> nodes. >>>>>>> >>>>>>>> Via "VM-VM affinity rules", anti-affinity is also configured. >>>>>>>>>>> >>>>>>>>>>> But >>>>>>>>>> >>>>>>>>> there >>>>>> >>>>>>> is >>>>>>>>> >>>>>>>>>> a single set of rule definition for both affinity (similar to >>>>>>>>>>> >>>>>>>>>>> locality >>>>>>>>>> >>>>>>>>> in >>>>>>>>> >>>>>>>>> our case) and anti-affinity. >>>>>>>>>> >>>>>>>>>>> Would it be a better approach for configuring locality rules >>>>>>>>>>> >>>>>>>>>>> and >>>>>>>>>> >>>>>>>>> anti-affinity rules in a single rule and call it "affinity >>>>>> >>>>>>> rule". >>>>>>>>>> >>>>>>>>> Thanks, >>>>>> >>>>>>> Chinmay. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 22, 2016 at 12:24 PM, Yogi Devendra < >>>>>>>>>>> >>>>>>>>>>> [email protected] >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> @Isha >>>>>>>>>>> >>>>>>>>>>>> I understand that anti-affinity across application is not >>>>>>>>>>>> >>>>>>>>>>>> straight-forward. >>>>>>>>>>> >>>>>>>>>>> It would be OK even if we do not have it in iteration 1. >>>>>>>>>>>> >>>>>>>>>>>> But, for attributes syntax; I still think that Java object >>>>>>>>>>>> >>>>>>>>>>>> should >>>>>>>>>>> >>>>>>>>>> be >>>>>>> >>>>>>>> avoided as they will be hard to configure from dt-site.xml or >>>>>>>>> >>>>>>>>>> other >>>>>>>>>>> >>>>>>>>>> config >>>>>>>> >>>>>>>>> files. >>>>>>>>>>>> Other suggestion for this could be JSON representation of >>>>>>>>>>>> >>>>>>>>>>>> String >>>>>>>>>>> >>>>>>>>>> array: >>>>>>> >>>>>>>> ["O2", "O3"]. (If operator names has some special characters >>>>>>>>>> >>>>>>>>>>> like >>>>>>>>>>> >>>>>>>>>> " >>>>>>>> >>>>>>>> or [ >>>>>>>>> >>>>>>>>>> or , those will be escaped in the JSON representation.) >>>>>>>>>>> >>>>>>>>>>>> Not sure if others agree on this; but attribute syntax should >>>>>>>>>>>> >>>>>>>>>>>> be >>>>>>>>>>> >>>>>>>>>> finalized >>>>>>> >>>>>>>> in iteration 1 to avoid backward compatibility issues later. >>>>>>>>>>>> >>>>>>>>>>>> ~ Yogi >>>>>>>>>>>> >>>>>>>>>>>> On 22 January 2016 at 00:43, Thomas Weise < >>>>>>>>>>>> >>>>>>>>>>>> [email protected]> >>>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>> >>>>>>>>> Node based requests is the best approach - if it works :-) >>>>>>>>>>>> >>>>>>>>>>>>> Blacklisting will require to allocate the containers >>>>>>>>>>>>> >>>>>>>>>>>>> sequentially. >>>>>>>>>>>> >>>>>>>>>>> It >>>>>>>>> >>>>>>>>> will >>>>>>>>>> >>>>>>>>>>> work, but slow down application startup, especially for >>>>>>>>>>>>> >>>>>>>>>>>>> larger >>>>>>>>>>>> >>>>>>>>>>> topologies. >>>>>>> >>>>>>>> On Thu, Jan 21, 2016 at 10:42 AM, Isha Arkatkar < >>>>>>>>>>>>> >>>>>>>>>>>>> [email protected]> >>>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>>> Should we consider the node based requests if it works >>>>>>>>>>>>>> >>>>>>>>>>>>>> with >>>>>>>>>>>>> >>>>>>>>>>>> Capacity >>>>>>> >>>>>>>> Scheduler or avoid 2b approach altogether? I checked that >>>>>>>>>>>> >>>>>>>>>>>>> node >>>>>>>>>>>>> >>>>>>>>>>>> requests >>>>>>>> >>>>>>>>> do >>>>>>>>>>>> >>>>>>>>>>>>> not work with fair scheduler on CDH cluster. Yarn does >>>>>>>>>>>>>> >>>>>>>>>>>>>> not >>>>>>>>>>>>> >>>>>>>>>>>> return >>>>>> >>>>>>> any >>>>>>>>> >>>>>>>>>> container if hostname is given in the container request. >>>>>>>>>>> >>>>>>>>>>>> I >>>>>>>>>>>>> >>>>>>>>>>>> am >>>>>> >>>>>> trying >>>>>>> >>>>>>>> to >>>>>>>>>>> >>>>>>>>>>> setup a small virtual hortonworks cluster to check the >>>>>>>>>>>> >>>>>>>>>>>>> this >>>>>>>>>>>>> >>>>>>>>>>>> behavior >>>>>> >>>>>>> on >>>>>>>>>>> >>>>>>>>>>> that. >>>>>>>>>>>> >>>>>>>>>>>>> YARN-2027 < >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/YARN-2027 >>>>>>>>>>>>> >>>>>>>>>>>> mentioned >>>>>> >>>>>>> that >>>>>>>>>>>> >>>>>>>>>>>>> container requests are not honored in capacity scheduler >>>>>>>>>>>>>> >>>>>>>>>>>>>> too. >>>>>>>>>>>>> >>>>>>>>>>>> But I >>>>>>> >>>>>>>> am >>>>>>>>>> >>>>>>>>>>> not >>>>>>>>>>>> >>>>>>>>>>>>> sure if it is because of distro dependent issue. Please >>>>>>>>>>>>>> >>>>>>>>>>>>>> share >>>>>>>>>>>>> >>>>>>>>>>>> insights. >>>>>>> >>>>>>>> @Vlad, Adding support for regular expression sounds good. >>>>>>>>>>>> >>>>>>>>>>>>> We >>>>>>>>>>>>> >>>>>>>>>>>> could >>>>>>> >>>>>>>> translate to list of operator names internally based on >>>>>>>>>> >>>>>>>>>>> regex. >>>>>>>>>>>>> >>>>>>>>>>>> @Yogi, I went with a list of strings for attribute >>>>>>>> >>>>>>>>> because >>>>>>>>>>>>> >>>>>>>>>>>> "O2, >>>>>> >>>>>>> O3" >>>>>>>>> >>>>>>>>>> could >>>>>>>>>>> >>>>>>>>>>>> be a valid single operator name too :) >>>>>>>>>>>>>> I am not sure of ways to implement anti-affinity across >>>>>>>>>>>>>> >>>>>>>>>>>>>> application. >>>>>>>>>>>>> >>>>>>>>>>>> Though >>>>>>>>>>> >>>>>>>>>>>> something to consider for later iteration. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Isha >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:59 PM, Thomas Weise < >>>>>>>>>>>>>> >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/SLIDER-82 >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 8:56 PM, Thomas Weise < >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> The point was that containers are taken away from >>>>>>>>>>>>>>> other >>>>>>>>>>>>>>> >>>>>>>>>>>>>> apps >>>>>> >>>>>>> that >>>>>>>>> >>>>>>>>>> may >>>>>>>>>>> >>>>>>>>>>>> have >>>>>>>>>>>>> >>>>>>>>>>>>>> to discard work etc. It's not good style to claim >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> resources >>>>>>>>>>>>>>> >>>>>>>>>>>>>> and >>>>>>>> >>>>>>>>> not >>>>>>>>>> >>>>>>>>>>> use >>>>>>>>>>>> >>>>>>>>>>>>> them eventually :-) >>>>>>>>>>>>>> >>>>>>>>>>>>>>> For this feature it is necessary to look at the >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> scheduler >>>>>>>>>>>>>>> >>>>>>>>>>>>>> capabilities/semantics and limitations. For example, >>>>>>> >>>>>>>> don't >>>>>>>>>>>>>>> >>>>>>>>>>>>>> bet >>>>>>>> >>>>>>>>> exclusively >>>>>>>>>> >>>>>>>>>>> on node requests if the goal is for it to work with >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> FairScheduler. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Also look at Slider, which just recently added >>>>>>>>>>>> >>>>>>>>>>>>> support >>>>>>>>>>>>>>> >>>>>>>>>>>>>> for >>>>>> >>>>>>> anti-affinity >>>>>>>> >>>>>>>>> (using node requests). When you run it on the CDH >>>>>>>>>>>>>>> cluster, >>>>>>>>>>>>>>> >>>>>>>>>>>>>> it >>>>>>>> >>>>>>>> probably >>>>>>>>> >>>>>>>>>> won't work... >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Jan 20, 2016 at 3:19 PM, Pramod Immaneni < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Once released won't the containers be available >>>>>>>>>>>>>>>> again >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> in >>>>>> >>>>>> the >>>>>>> >>>>>>>> pool. >>>>>>>>> >>>>>>>>>> This >>>>>>>>>>>> >>>>>>>>>>>>> would only be optional and not mandatory. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Jan 19, 2016 at 2:02 PM, Thomas Weise < >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> How about also supporting a minor variation of it >>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> an >>>>>> >>>>>>> option >>>>>>>> >>>>>>>>> where it greedily gets the total number of >>>>>>>>>>> >>>>>>>>>>>> containers >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> and >>>>>>>> >>>>>>>>> discards >>>>>>>>>> >>>>>>>>>>> ones >>>>>>>>>>>>>> >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> can't use and repeats the process for the >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> remaining >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> till >>>>>>> >>>>>>>> everything >>>>>>>>> >>>>>>>>>> has >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> been allocated. >>>>>>>>>>>>>>>>>> This is problematic as with resource preemption >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> containers >>>>>>> >>>>>>>> will >>>>>>>>>>>> >>>>>>>>>>>>> be >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> potentially taken away from other applications and >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> thrown >>>>>>>> >>>>>>>>> away. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Also does it make sense to support anti-cluster >>>>>>>>>>>>>>>>>> affinity? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>>> On Tue, Jan 19, 2016 at 1:21 PM, Isha Arkatkar < >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> We want add support for Anti-affinity in >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Apex >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> to >>>>>> >>>>>>> allow >>>>>>>> >>>>>>>>> applications >>>>>>>>>>> >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> launch specific physical operators on >>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> nodes(APEXCORE-10 >>>>>> >>>>>>> < >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/APEXCORE-10 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ). >>>>>>>> >>>>>>>>> Want >>>>>>>>>> >>>>>>>>>>> to >>>>>>>>>>>> >>>>>>>>>>>> request >>>>>>>>>>>>> >>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> suggestions/ideas for the same! >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The reasons for using anti-affinity in >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> operators >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> could >>>>>>> >>>>>>>> be: >>>>>>>>>> >>>>>>>>>>> to >>>>>>>>>>>> >>>>>>>>>>>>> ensure >>>>>>>>>>>>>> >>>>>>>>>>>>>>> reliability, for performance reasons (such as >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> application >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> may >>>>>>>>>>> >>>>>>>>>>>> not >>>>>>>>>>>>> >>>>>>>>>>>>>> want >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> i/o intensive operators to land on the same >>>>>>>>>>>>>>>>>>> node >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> to >>>>>> >>>>>>> improve >>>>>>>> >>>>>>>>> performance) >>>>>>>>>>>> >>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> for some application specific constraints(for >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> example, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2 >>>>>>>>> >>>>>>>>>> partitions >>>>>>>>>>> >>>>>>>>>>>> cannot >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> be run on the same node since they use same >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> port >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> number). >>>>>> >>>>>>> This >>>>>>>>>>> >>>>>>>>>>>> is >>>>>>>>>>>>> >>>>>>>>>>>>>> the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> general rationale for adding Anti-affinity >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> support. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Since, Yarn does not support anti-affinity yet >>>>>>>> >>>>>>>>> (YARN-1042 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> < >>>>>>>>>>> >>>>>>>>>>>> https://issues.apache.org/jira/browse/YARN-1042 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ), >>>>>>> >>>>>>>> we >>>>>>>>> >>>>>>>>> need >>>>>>>>>> >>>>>>>>>>> to >>>>>>>>>>>> >>>>>>>>>>>>> implement >>>>>>>>>>>>>> >>>>>>>>>>>>>>> the logic in AM. Wanted to get your views on >>>>>>>>>>>>>>>>>>> following >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> aspects >>>>>>>>> >>>>>>>>>> for >>>>>>>>>>>>> >>>>>>>>>>>>>> this >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> implementation: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> *1. How to specify anti-affinity for physical >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> operators/partitions >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> application:* >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> One way for this is to have an attribute >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> setting >>>>>> >>>>>>> anti-affinity >>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> the logical operator context. And an operator >>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> set >>>>>>> >>>>>>>> this >>>>>>>>> >>>>>>>>>> attribute >>>>>>>>>>> >>>>>>>>>>>> with >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> list of operator names which should not be >>>>>>>>>>>>>>>>>>> collocated. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Consider dag with 3 operators: >>>>>>>>> >>>>>>>>>> TestOperator o1 = dag.addOperator("O1", >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> TestOperator()); >>>>>> >>>>>>> TestOperator o2 = dag.addOperator("O2", >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> TestOperator()); >>>>>> >>>>>>> TestOperator o3 = dag.addOperator("O3", >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> TestOperator()); >>>>>> >>>>>>> To set anti-affinity for O1 operator: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> dag.setAttribute(o1, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> OperatorContext.ANTI_AFFINITY, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> new >>>>>>>>>> >>>>>>>>>>> ArrayList<String>(Arrays.asList("O2", "O3"))); >>>>>>>>>>>> >>>>>>>>>>>>> This would mean O1 should not be >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> allocated >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> on >>>>>> >>>>>> nodes >>>>>>> >>>>>>>> containing >>>>>>>>>> >>>>>>>>>>> operators O2 and O3. This applies to all >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> allocated >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> partitions >>>>>>> >>>>>>>> of >>>>>>>>>>>>> >>>>>>>>>>>>> O1, >>>>>>>>>>>>>> >>>>>>>>>>>>>>> O2, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> O3. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Also, if same operator name is part of >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> anti-affinity >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> list, >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >
