Re: Re: does spark really support label expr like && or || ?

On Wed, Dec 16, 2015 at 6:31 PM, Allen Zhang wrote: > so , my question is does the spark.yarn.executor.nodeLabelExpression and > spark.yarn.am.nodeLabelExpression really support "EXPRESSION" like and &&, > or ||, or even ! and so on. Spark doesn't do anything with those

Re: does spark really support label expr like && or || ?

more details commands: 2. yarn rmadmin -replaceLabelsOnNode spark-dev:54321,foo; yarn rmadmin -replaceLabelsOnNode sut-1:54321,bar; yarn rmadmin -replaceLabelsOnNode sut-2:54321,bye; yarn rmadmin -replaceLabelsOnNode sut-3:54321,foo; At 2015-12-17 10:31:20, "Allen Zhang"

Re: does spark really support label expr like && or || ?

are you trying to do dataframe boolean expression? please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions. example: >>> df = sqlContext.range(10) >>> df.where( (df.id==1) | ~(df.id==1)) DataFrame[id: bigint] On Wed, Dec 16, 2015 at 4:32 PM, Allen

Re: does spark really support label expr like && or || ?

Allen: Since you mentioned scheduling, I assume you were talking about node label support in YARN. If that is the case, can you give us some more information: How node labels are setup in YARN cluster How you specified node labels in application Hadoop and Spark releases you are using Cheers >