I tried with condition expression also but it didn't work :( On Aug 8, 2016 11:13 AM, "Chanh Le" <giaosu...@gmail.com> wrote:
> You should use *df.where(conditionExpr)* which is more convenient to > express some simple term in SQL. > > > /** > * Filters rows using the given SQL expression. > * {{{ > * peopleDf.where("age > 15") > * }}} > * @group dfops > * @since 1.5.0 > */ > def where(conditionExpr: String): DataFrame = { > filter(Column(SqlParser.parseExpression(conditionExpr))) > } > > > > > > On Aug 7, 2016, at 10:58 PM, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > although the logic should be col1 <> a && col(1) <> b > > to exclude both > > Like > > df.filter('transactiontype > " ").filter(not('transactiontype ==="DEB") && > not('transactiontype ==="BGC")).select('transactiontype).distinct. > collect.foreach(println) > > HTH > > Dr Mich Talebzadeh > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > http://talebzadehmich.wordpress.com > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > On 7 August 2016 at 16:53, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> try similar to this >> >> df.filter(not('transactiontype ==="DEB") || not('transactiontype >> ==="CRE")) >> >> HTH >> >> >> Dr Mich Talebzadeh >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> http://talebzadehmich.wordpress.com >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 7 August 2016 at 15:43, Divya Gehlot <divya.htco...@gmail.com> wrote: >> >>> Hi, >>> I have use case where I need to use or[||] operator in filter condition. >>> It seems its not working its taking the condition before the operator >>> and ignoring the other filter condition after or operator. >>> As any body faced similar issue . >>> >>> Psuedo code : >>> df.filter(col("colName").notEqual("no_value") || >>> col("colName").notEqual("")) >>> >>> Am I missing something. >>> Would really appreciate the help. >>> >>> >>> Thanks, >>> Divya >>> >> >> > >