You should use df.where(conditionExpr) which is more convenient to express some simple term in SQL.
/** * Filters rows using the given SQL expression. * {{{ * peopleDf.where("age > 15") * }}} * @group dfops * @since 1.5.0 */ def where(conditionExpr: String): DataFrame = { filter(Column(SqlParser.parseExpression(conditionExpr))) } > On Aug 7, 2016, at 10:58 PM, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > although the logic should be col1 <> a && col(1) <> b > > to exclude both > > Like > > df.filter('transactiontype > " ").filter(not('transactiontype ==="DEB") && > not('transactiontype > ==="BGC")).select('transactiontype).distinct.collect.foreach(println) > > HTH > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw> > > http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> > > Disclaimer: Use it at your own risk. Any and all responsibility for any loss, > damage or destruction of data or any other property which may arise from > relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > On 7 August 2016 at 16:53, Mich Talebzadeh <mich.talebza...@gmail.com > <mailto:mich.talebza...@gmail.com>> wrote: > try similar to this > > df.filter(not('transactiontype ==="DEB") || not('transactiontype ==="CRE")) > > HTH > > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw> > > http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> > > Disclaimer: Use it at your own risk. Any and all responsibility for any loss, > damage or destruction of data or any other property which may arise from > relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > On 7 August 2016 at 15:43, Divya Gehlot <divya.htco...@gmail.com > <mailto:divya.htco...@gmail.com>> wrote: > Hi, > I have use case where I need to use or[||] operator in filter condition. > It seems its not working its taking the condition before the operator and > ignoring the other filter condition after or operator. > As any body faced similar issue . > > Psuedo code : > df.filter(col("colName").notEqual("no_value") || col("colName").notEqual("")) > > Am I missing something. > Would really appreciate the help. > > > Thanks, > Divya > >