Re: [Spark1.6] Or (||) operator not working in DataFrame

I am afraid logic is incorrect. that is the reason why it is not working. Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: [Spark1.6] Or (||) operator not working in DataFrame

I tried with condition expression also but it didn't work :( On Aug 8, 2016 11:13 AM, "Chanh Le" wrote: > You should use *df.where(conditionExpr)* which is more convenient to > express some simple term in SQL. > > > /** > * Filters rows using the given SQL expression. >

Re: [Spark1.6] Or (||) operator not working in DataFrame

You should use df.where(conditionExpr) which is more convenient to express some simple term in SQL. /** * Filters rows using the given SQL expression. * {{{ * peopleDf.where("age > 15") * }}} * @group dfops * @since 1.5.0 */ def where(conditionExpr: String): DataFrame = {

Re: [Spark1.6] Or (||) operator not working in DataFrame

although the logic should be col1 <> a && col(1) <> b to exclude both Like df.filter('transactiontype > " ").filter(not('transactiontype ==="DEB") && not('transactiontype ==="BGC")).select('transactiontype).distinct.collect.foreach(println) HTH Dr Mich Talebzadeh LinkedIn *

Re: [Spark1.6] Or (||) operator not working in DataFrame

try similar to this df.filter(not('transactiontype ==="DEB") || not('transactiontype ==="CRE")) HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: [Spark1.6] Or (||) operator not working in DataFrame

Can you try 'or' keyword instead? On Aug 7, 2016 7:43 AM, "Divya Gehlot" wrote: > Hi, > I have use case where I need to use or[||] operator in filter condition. > It seems its not working its taking the condition before the operator and > ignoring the other filter

[Spark1.6] Or (||) operator not working in DataFrame

Hi, I have use case where I need to use or[||] operator in filter condition. It seems its not working its taking the condition before the operator and ignoring the other filter condition after or operator. As any body faced similar issue . Psuedo code :