Re: [Spark1.6] Or (||) operator not working in DataFrame

[Mich Talebzadeh]

I am afraid logic is incorrect. that is the reason why it is not working.

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 8 August 2016 at 04:36, Divya Gehlot  wrote:

> I tried with condition expression  also but it didn't work :(
>
> On Aug 8, 2016 11:13 AM, "Chanh Le"  wrote:
>
>> You should use *df.where(conditionExpr)* which is more convenient to
>> express some simple term in SQL.
>>
>>
>> /**
>>  * Filters rows using the given SQL expression.
>>  * {{{
>>  *   peopleDf.where("age > 15")
>>  * }}}
>>  * @group dfops
>>  * @since 1.5.0
>>  */
>> def where(conditionExpr: String): DataFrame = {
>>   filter(Column(SqlParser.parseExpression(conditionExpr)))
>> }
>>
>>
>>
>>
>>
>> On Aug 7, 2016, at 10:58 PM, Mich Talebzadeh 
>> wrote:
>>
>> although the logic should be col1 <> a && col(1) <> b
>>
>> to exclude both
>>
>> Like
>>
>> df.filter('transactiontype > " ").filter(not('transactiontype ==="DEB")
>> && not('transactiontype ==="BGC")).select('transaction
>> type).distinct.collect.foreach(println)
>>
>> HTH
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 7 August 2016 at 16:53, Mich Talebzadeh 
>> wrote:
>>
>>> try similar to this
>>>
>>> df.filter(not('transactiontype ==="DEB") || not('transactiontype
>>> ==="CRE"))
>>>
>>> HTH
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 7 August 2016 at 15:43, Divya Gehlot  wrote:
>>>
 Hi,
 I have use case where I need to use or[||] operator in filter condition.
 It seems its not working its taking the condition before the operator
 and ignoring the other filter condition after or operator.
 As any body faced similar issue .

 Psuedo code :
 df.filter(col("colName").notEqual("no_value") ||
 col("colName").notEqual(""))

 Am I missing something.
 Would really appreciate the help.


 Thanks,
 Divya

>>>
>>>
>>
>>

Re: [Spark1.6] Or (||) operator not working in DataFrame

[Divya Gehlot]

I tried with condition expression  also but it didn't work :(

On Aug 8, 2016 11:13 AM, "Chanh Le"  wrote:

> You should use *df.where(conditionExpr)* which is more convenient to
> express some simple term in SQL.
>
>
> /**
>  * Filters rows using the given SQL expression.
>  * {{{
>  *   peopleDf.where("age > 15")
>  * }}}
>  * @group dfops
>  * @since 1.5.0
>  */
> def where(conditionExpr: String): DataFrame = {
>   filter(Column(SqlParser.parseExpression(conditionExpr)))
> }
>
>
>
>
>
> On Aug 7, 2016, at 10:58 PM, Mich Talebzadeh 
> wrote:
>
> although the logic should be col1 <> a && col(1) <> b
>
> to exclude both
>
> Like
>
> df.filter('transactiontype > " ").filter(not('transactiontype ==="DEB") &&
> not('transactiontype ==="BGC")).select('transactiontype).distinct.
> collect.foreach(println)
>
> HTH
>
> Dr Mich Talebzadeh
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
> http://talebzadehmich.wordpress.com
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 7 August 2016 at 16:53, Mich Talebzadeh 
> wrote:
>
>> try similar to this
>>
>> df.filter(not('transactiontype ==="DEB") || not('transactiontype
>> ==="CRE"))
>>
>> HTH
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 7 August 2016 at 15:43, Divya Gehlot  wrote:
>>
>>> Hi,
>>> I have use case where I need to use or[||] operator in filter condition.
>>> It seems its not working its taking the condition before the operator
>>> and ignoring the other filter condition after or operator.
>>> As any body faced similar issue .
>>>
>>> Psuedo code :
>>> df.filter(col("colName").notEqual("no_value") ||
>>> col("colName").notEqual(""))
>>>
>>> Am I missing something.
>>> Would really appreciate the help.
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>
>

Re: [Spark1.6] Or (||) operator not working in DataFrame

[Chanh Le]

You should use df.where(conditionExpr) which is more convenient to express some 
simple term in SQL.
 

/**
 * Filters rows using the given SQL expression.
 * {{{
 *   peopleDf.where("age > 15")
 * }}}
 * @group dfops
 * @since 1.5.0
 */
def where(conditionExpr: String): DataFrame = {
  filter(Column(SqlParser.parseExpression(conditionExpr)))
}




> On Aug 7, 2016, at 10:58 PM, Mich Talebzadeh  
> wrote:
> 
> although the logic should be col1 <> a && col(1) <> b
> 
> to exclude both
> 
> Like
> 
> df.filter('transactiontype > " ").filter(not('transactiontype ==="DEB") && 
> not('transactiontype 
> ==="BGC")).select('transactiontype).distinct.collect.foreach(println)
> 
> HTH
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
> On 7 August 2016 at 16:53, Mich Talebzadeh  > wrote:
> try similar to this
> 
> df.filter(not('transactiontype ==="DEB") || not('transactiontype ==="CRE"))
> 
> HTH
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
> On 7 August 2016 at 15:43, Divya Gehlot  > wrote:
> Hi,
> I have use case where I need to use or[||] operator in filter condition.
> It seems its not working its taking the condition before the operator and 
> ignoring the other filter condition after or operator.
> As any body faced similar issue .
> 
> Psuedo code :
> df.filter(col("colName").notEqual("no_value") || col("colName").notEqual(""))
> 
> Am I missing something.
> Would really appreciate the help.
> 
> 
> Thanks,
> Divya 
> 
> 

Re: [Spark1.6] Or (||) operator not working in DataFrame

[Mich Talebzadeh]

although the logic should be col1 <> a && col(1) <> b

to exclude both

Like

df.filter('transactiontype > " ").filter(not('transactiontype ==="DEB") &&
not('transactiontype
==="BGC")).select('transactiontype).distinct.collect.foreach(println)

HTH

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 7 August 2016 at 16:53, Mich Talebzadeh 
wrote:

> try similar to this
>
> df.filter(not('transactiontype ==="DEB") || not('transactiontype ==="CRE"))
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 7 August 2016 at 15:43, Divya Gehlot  wrote:
>
>> Hi,
>> I have use case where I need to use or[||] operator in filter condition.
>> It seems its not working its taking the condition before the operator and
>> ignoring the other filter condition after or operator.
>> As any body faced similar issue .
>>
>> Psuedo code :
>> df.filter(col("colName").notEqual("no_value") ||
>> col("colName").notEqual(""))
>>
>> Am I missing something.
>> Would really appreciate the help.
>>
>>
>> Thanks,
>> Divya
>>
>
>

Re: [Spark1.6] Or (||) operator not working in DataFrame

[Mich Talebzadeh]

try similar to this

df.filter(not('transactiontype ==="DEB") || not('transactiontype ==="CRE"))

HTH


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 7 August 2016 at 15:43, Divya Gehlot  wrote:

> Hi,
> I have use case where I need to use or[||] operator in filter condition.
> It seems its not working its taking the condition before the operator and
> ignoring the other filter condition after or operator.
> As any body faced similar issue .
>
> Psuedo code :
> df.filter(col("colName").notEqual("no_value") ||
> col("colName").notEqual(""))
>
> Am I missing something.
> Would really appreciate the help.
>
>
> Thanks,
> Divya
>

Re: [Spark1.6] Or (||) operator not working in DataFrame

[janardhan shetty]

Can you try 'or' keyword instead?
On Aug 7, 2016 7:43 AM, "Divya Gehlot"  wrote:

> Hi,
> I have use case where I need to use or[||] operator in filter condition.
> It seems its not working its taking the condition before the operator and
> ignoring the other filter condition after or operator.
> As any body faced similar issue .
>
> Psuedo code :
> df.filter(col("colName").notEqual("no_value") ||
> col("colName").notEqual(""))
>
> Am I missing something.
> Would really appreciate the help.
>
>
> Thanks,
> Divya
>

[Spark1.6] Or (||) operator not working in DataFrame

[Divya Gehlot]

Hi,
I have use case where I need to use or[||] operator in filter condition.
It seems its not working its taking the condition before the operator and
ignoring the other filter condition after or operator.
As any body faced similar issue .

Psuedo code :
df.filter(col("colName").notEqual("no_value") ||
col("colName").notEqual(""))

Am I missing something.
Would really appreciate the help.


Thanks,
Divya