Re: Spark 2.0 - Join statement compile error

[shengshanzhang]

Hi ,try this way

val df = sales_demand.join(product_master,sales_demand("INVENTORY_ITEM_ID") === 
product_master("INVENTORY_ITEM_ID"),"inner")

> 在 2016年8月30日，下午5:52，Jacek Laskowski  写道：
> 
> Hi Mich,
> 
> This is the first time I've been told about $ for string interpolation (as 
> the function not the placeholder). Thanks for letting me know about it!
> 
> What is often used is s"whatever you want to reference inside the string 
> $-prefix unless it is a complex expression" i.e.
> 
> scala> s"I'm using $spark in ${spark.version}"
> res0: String = I'm using org.apache.spark.sql.SparkSession@1fc1c7e in 
> 2.1.0-SNAPSHOT
> 
> 
> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/ 
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark 
> 
> Follow me at https://twitter.com/jaceklaskowski 
> 
> On Tue, Aug 30, 2016 at 10:27 AM, Mich Talebzadeh  > wrote:
> Actually I doubled checked this ‘s’ String Interpolator
> 
> In Scala
> 
> scala> val chars =  "This is Scala"
> chars: String = This is Scala
> scala> println($"$chars")
> This is Scala
> 
> OK so far fine. In shell (ksh) can do
> 
> chars="This is Scala"
> print "$chars"
> This is Scala
> 
> In Shell
> 
> print "$charsand it is interesting"
>  it is interesting
> 
> Have a problem with interpretation!
> 
> This will work by using {} brackers
> 
> print "${chars} and it is interesting"
> This is Scala and it is interesting
> 
> 
> In Scala
> 
> println($"$charsand it is interesting")
> :24: error: not found: value charsand
>println($"$charsand it is interesting")
> 
> Likewise shell this will work by deploying the brackets {}
> println($"${chars} and it is interesting")
> This is Scala and it is interesting
> 
> So I presume it is best practice to use ${} in Scala like shell? Although in 
> most cases $VALUE should work.
> 
> 
> Cheers
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
> On 28 August 2016 at 09:25, Mich Talebzadeh  > wrote:
> Yes I realised that. Actually I thought it was s not $. it has been around in 
> shell for years say for actual values --> ${LOG_FILE}, for position 's/ etc
> 
> 
> cat ${LOG_FILE} | egrep -v 'rows affected|return status|&&&' | sed -e 's/^[   
>  ]*//g' -e 's/^//g' -e '/^$/d' > temp.out
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
> 
> Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
> damage or destruction of data or any other property which may arise from 
> relying on this email's technical content is explicitly disclaimed. The 
> author will in no case be liable for any monetary damages arising from such 
> loss, damage or destruction.
>  
> 
> On 28 August 2016 at 08:47, Jacek Laskowski  > wrote:
> Hi Mich,
> 
> This is Scala's string interpolation which allow for replacing $-prefixed 
> expressions with their values.
> 
> It's what cool kids use in Scala to do templating and concatenation 
> 
> Jacek
> 
> 
> On 23 Aug 2016 9:21 a.m., "Mich Talebzadeh"  > wrote:
> What is   --> s below before the text of sql?
> 
> var sales_order_sql_stmt = s"""SELECT ORDER_NUMBER , INVENTORY_ITEM_ID, 
> ORGANIZATION_ID,
> 
>   from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd'), 
> '-MM-dd') AS schedule_date
> 
>   FROM sales_order_demand
> 
>   WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd') >= 
> $planning_start_date  limit 10"""
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> 
>  
> http://talebzadehmich.wordpress.com 
> 

Re: Spark 2.0 - Join statement compile error

[Jacek Laskowski]

Hi Mich,

This is the first time I've been told about $ for string interpolation (as
the function not the placeholder). Thanks for letting me know about it!

What is often used is s"whatever you want to reference inside the string
$-prefix unless it is a complex expression" i.e.

scala> s"I'm using $spark in ${spark.version}"
res0: String = I'm using org.apache.spark.sql.SparkSession@1fc1c7e in
2.1.0-SNAPSHOT


Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

On Tue, Aug 30, 2016 at 10:27 AM, Mich Talebzadeh  wrote:

> Actually I doubled checked this ‘s’ String Interpolator
>
> In Scala
>
> scala> val chars =  "This is Scala"
> chars: String = This is Scala
>
> scala> println($"$chars")
> This is Scala
> OK so far fine. In shell (ksh) can do
>
> chars="This is Scala"
> print "$chars"
> This is Scala
>
> In Shell
>
> print "$charsand it is interesting"
>  it is interesting
>
> Have a problem with interpretation!
>
> This will work by using {} brackers
>
> print "${chars} and it is interesting"
> This is Scala and it is interesting
>
>
> In Scala
>
> println($"$charsand it is interesting")
> :24: error: not found: value charsand
>println($"$charsand it is interesting")
>
> Likewise shell this will work by deploying the brackets {}
>
> println($"${chars} and it is interesting")
> This is Scala and it is interesting
> So I presume it is best practice to use ${} in Scala like shell? Although
> in most cases $VALUE should work.
>
>
> Cheers
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 28 August 2016 at 09:25, Mich Talebzadeh 
> wrote:
>
>> Yes I realised that. Actually I thought it was s not $. it has been
>> around in shell for years say for actual values --> ${LOG_FILE}, for
>> position 's/ etc
>>
>>
>> cat ${LOG_FILE} | egrep -v 'rows affected|return status|&&&' | sed -e
>> 's/^[]*//g' -e 's/^//g' -e '/^$/d' > temp.out
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 28 August 2016 at 08:47, Jacek Laskowski  wrote:
>>
>>> Hi Mich,
>>>
>>> This is Scala's string interpolation which allow for replacing
>>> $-prefixed expressions with their values.
>>>
>>> It's what cool kids use in Scala to do templating and concatenation 
>>>
>>> Jacek
>>>
>>> On 23 Aug 2016 9:21 a.m., "Mich Talebzadeh" 
>>> wrote:
>>>
 What is   --> s below before the text of sql?

 *var* sales_order_sql_stmt =* s*"""SELECT ORDER_NUMBER ,
 INVENTORY_ITEM_ID, ORGANIZATION_ID,

   from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd'),
 '-MM-dd') AS schedule_date

   FROM sales_order_demand

   WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd') >= $
 planning_start_date  limit 10"""

 Dr Mich Talebzadeh



 LinkedIn * 
 https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
 *



 http://talebzadehmich.wordpress.com


 *Disclaimer:* Use it at your own risk. Any and all responsibility for
 any loss, damage or destruction of data or any other property which may
 arise from relying on this email's technical content is explicitly
 disclaimed. The author will in no case be liable for any monetary damages
 arising from such loss, damage or destruction.



 On 23 August 2016 at 07:31, Deepak Sharma 
 wrote:

>
> On Tue, Aug 23, 2016 at 10:32 AM, Deepak Sharma  > wrote:
>
>> 

Re: Spark 2.0 - Join statement compile error

[Mich Talebzadeh]

Actually I doubled checked this ‘s’ String Interpolator

In Scala

scala> val chars =  "This is Scala"
chars: String = This is Scala

scala> println($"$chars")
This is Scala
OK so far fine. In shell (ksh) can do

chars="This is Scala"
print "$chars"
This is Scala

In Shell

print "$charsand it is interesting"
 it is interesting

Have a problem with interpretation!

This will work by using {} brackers

print "${chars} and it is interesting"
This is Scala and it is interesting


In Scala

println($"$charsand it is interesting")
:24: error: not found: value charsand
   println($"$charsand it is interesting")

Likewise shell this will work by deploying the brackets {}

println($"${chars} and it is interesting")
This is Scala and it is interesting
So I presume it is best practice to use ${} in Scala like shell? Although
in most cases $VALUE should work.


Cheers





















Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 28 August 2016 at 09:25, Mich Talebzadeh 
wrote:

> Yes I realised that. Actually I thought it was s not $. it has been around
> in shell for years say for actual values --> ${LOG_FILE}, for position 's/
> etc
>
>
> cat ${LOG_FILE} | egrep -v 'rows affected|return status|&&&' | sed -e
> 's/^[]*//g' -e 's/^//g' -e '/^$/d' > temp.out
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 28 August 2016 at 08:47, Jacek Laskowski  wrote:
>
>> Hi Mich,
>>
>> This is Scala's string interpolation which allow for replacing $-prefixed
>> expressions with their values.
>>
>> It's what cool kids use in Scala to do templating and concatenation 
>>
>> Jacek
>>
>> On 23 Aug 2016 9:21 a.m., "Mich Talebzadeh" 
>> wrote:
>>
>>> What is   --> s below before the text of sql?
>>>
>>> *var* sales_order_sql_stmt =* s*"""SELECT ORDER_NUMBER ,
>>> INVENTORY_ITEM_ID, ORGANIZATION_ID,
>>>
>>>   from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd'),
>>> '-MM-dd') AS schedule_date
>>>
>>>   FROM sales_order_demand
>>>
>>>   WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd') >= $
>>> planning_start_date  limit 10"""
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 23 August 2016 at 07:31, Deepak Sharma  wrote:
>>>

 On Tue, Aug 23, 2016 at 10:32 AM, Deepak Sharma 
 wrote:

> *val* *df** = **sales_demand**.**join**(**product_master**,*
> *sales_demand**.$"INVENTORY_ITEM_ID" =**== **product_master*
> *.$"INVENTORY_ITEM_ID",**"inner"**)*


 Ignore the last statement.
 It should look something like this:
 *val* *df** = **sales_demand**.**join**(**product_master**,$"*
 *sales_demand**.INVENTORY_ITEM_ID" =**== $"**product_master*
 *.INVENTORY_ITEM_ID",**"inner"**)*


 --
 Thanks
 Deepak
 www.bigdatabig.com
 www.keosha.net

>>>
>>>
>

Re: Spark 2.0 - Join statement compile error

[Mich Talebzadeh]

Yes I realised that. Actually I thought it was s not $. it has been around
in shell for years say for actual values --> ${LOG_FILE}, for position 's/
etc


cat ${LOG_FILE} | egrep -v 'rows affected|return status|&&&' | sed -e
's/^[]*//g' -e 's/^//g' -e '/^$/d' > temp.out


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 28 August 2016 at 08:47, Jacek Laskowski  wrote:

> Hi Mich,
>
> This is Scala's string interpolation which allow for replacing $-prefixed
> expressions with their values.
>
> It's what cool kids use in Scala to do templating and concatenation 
>
> Jacek
>
> On 23 Aug 2016 9:21 a.m., "Mich Talebzadeh" 
> wrote:
>
>> What is   --> s below before the text of sql?
>>
>> *var* sales_order_sql_stmt =* s*"""SELECT ORDER_NUMBER ,
>> INVENTORY_ITEM_ID, ORGANIZATION_ID,
>>
>>   from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd'),
>> '-MM-dd') AS schedule_date
>>
>>   FROM sales_order_demand
>>
>>   WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd') >= $
>> planning_start_date  limit 10"""
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>> On 23 August 2016 at 07:31, Deepak Sharma  wrote:
>>
>>>
>>> On Tue, Aug 23, 2016 at 10:32 AM, Deepak Sharma 
>>> wrote:
>>>
 *val* *df** = **sales_demand**.**join**(**product_master**,*
 *sales_demand**.$"INVENTORY_ITEM_ID" =**== **product_master*
 *.$"INVENTORY_ITEM_ID",**"inner"**)*
>>>
>>>
>>> Ignore the last statement.
>>> It should look something like this:
>>> *val* *df** = **sales_demand**.**join**(**product_master**,$"*
>>> *sales_demand**.INVENTORY_ITEM_ID" =**== $"**product_master*
>>> *.INVENTORY_ITEM_ID",**"inner"**)*
>>>
>>>
>>> --
>>> Thanks
>>> Deepak
>>> www.bigdatabig.com
>>> www.keosha.net
>>>
>>
>>

Re: Spark 2.0 - Join statement compile error

[Jacek Laskowski]

Hi Mich,

This is Scala's string interpolation which allow for replacing $-prefixed
expressions with their values.

It's what cool kids use in Scala to do templating and concatenation 

Jacek

On 23 Aug 2016 9:21 a.m., "Mich Talebzadeh" 
wrote:

> What is   --> s below before the text of sql?
>
> *var* sales_order_sql_stmt =* s*"""SELECT ORDER_NUMBER ,
> INVENTORY_ITEM_ID, ORGANIZATION_ID,
>
>   from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd'),
> '-MM-dd') AS schedule_date
>
>   FROM sales_order_demand
>
>   WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd') >= $
> planning_start_date  limit 10"""
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 23 August 2016 at 07:31, Deepak Sharma  wrote:
>
>>
>> On Tue, Aug 23, 2016 at 10:32 AM, Deepak Sharma 
>> wrote:
>>
>>> *val* *df** = **sales_demand**.**join**(**product_master**,*
>>> *sales_demand**.$"INVENTORY_ITEM_ID" =**== **product_master*
>>> *.$"INVENTORY_ITEM_ID",**"inner"**)*
>>
>>
>> Ignore the last statement.
>> It should look something like this:
>> *val* *df** = **sales_demand**.**join**(**product_master**,$"*
>> *sales_demand**.INVENTORY_ITEM_ID" =**== $"**product_master*
>> *.INVENTORY_ITEM_ID",**"inner"**)*
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>
>

Re: Spark 2.0 - Join statement compile error

[Mich Talebzadeh]

What is   --> s below before the text of sql?

*var* sales_order_sql_stmt =* s*"""SELECT ORDER_NUMBER , INVENTORY_ITEM_ID,
ORGANIZATION_ID,

  from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd'),
'-MM-dd') AS schedule_date

  FROM sales_order_demand

  WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd') >= $
planning_start_date  limit 10"""

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 23 August 2016 at 07:31, Deepak Sharma  wrote:

>
> On Tue, Aug 23, 2016 at 10:32 AM, Deepak Sharma 
> wrote:
>
>> *val* *df** = **sales_demand**.**join**(**product_master**,*
>> *sales_demand**.$"INVENTORY_ITEM_ID" =**== **product_master*
>> *.$"INVENTORY_ITEM_ID",**"inner"**)*
>
>
> Ignore the last statement.
> It should look something like this:
> *val* *df** = **sales_demand**.**join**(**product_master**,$"*
> *sales_demand**.INVENTORY_ITEM_ID" =**== $"**product_master*
> *.INVENTORY_ITEM_ID",**"inner"**)*
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>

Re: Spark 2.0 - Join statement compile error

[Deepak Sharma]

On Tue, Aug 23, 2016 at 10:32 AM, Deepak Sharma 
wrote:

> *val* *df** = 
> **sales_demand**.**join**(**product_master**,**sales_demand**.$"INVENTORY_ITEM_ID"
> =**== **product_master**.$"INVENTORY_ITEM_ID",**"inner"**)*


Ignore the last statement.
It should look something like this:
*val* *df** = 
**sales_demand**.**join**(**product_master**,$"**sales_demand**.INVENTORY_ITEM_ID"
=**== $"**product_master**.INVENTORY_ITEM_ID",**"inner"**)*


-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net

Re: Spark 2.0 - Join statement compile error

[Deepak Sharma]

Hi Subhajit
Try this in your join:
*val* *df** = 
**sales_demand**.**join**(**product_master**,**sales_demand**.$"INVENTORY_ITEM_ID"
=**== **product_master**.$"INVENTORY_ITEM_ID",**"inner"**)*

On Tue, Aug 23, 2016 at 2:30 AM, Subhajit Purkayastha 
wrote:

> *All,*
>
>
>
> *I have the following dataFrames and the temp table. *
>
>
>
> *I am trying to create a new DF , the following statement is not compiling*
>
>
>
> *val* *df** = **sales_demand**.**join**(**product_master**,(*
> *sales_demand**.INVENTORY_ITEM_ID**==**product_master*
> *.INVENTORY_ITEM_ID),**joinType**=**"inner"**)*
>
>
>
>
>
>
>
> *What am I doing wrong?*
>
>
>
> *==Code===*
>
>
>
> *var* sales_order_sql_stmt = s"""SELECT ORDER_NUMBER , INVENTORY_ITEM_ID,
> ORGANIZATION_ID,
>
>   from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd'),
> '-MM-dd') AS schedule_date
>
>   FROM sales_order_demand
>
>   WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd') >= $
> planning_start_date  limit 10"""
>
>
>
> *val* sales_demand = spark.sql (sales_order_sql_stmt)
>
>
>
> //print the data
>
> *sales_demand**.**collect**()*.foreach { println }
>
>
>
>
>
> *val* product_sql_stmt = "select 
> SEGMENT1,INVENTORY_ITEM_ID,ORGANIZATION_ID
> from product limit 10"
>
> *val* product_master = spark.sql (product_sql_stmt)
>
>
>
> //print the data
>
> *product_master**.**collect**()*.foreach { println }
>
>
>
> *val* *df** = **sales_demand**.**join**(**product_master**,(*
> *sales_demand**.INVENTORY_ITEM_ID**==**product_master*
> *.INVENTORY_ITEM_ID),**joinType**=**"inner"**)*
>
>
>
>
>
>
>
>spark.stop()
>



-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net

Re: Spark 2.0 - Join statement compile error

[Vishal Maru]

try putting join condition as String

On Mon, Aug 22, 2016 at 5:00 PM, Subhajit Purkayastha 
wrote:

> *All,*
>
>
>
> *I have the following dataFrames and the temp table. *
>
>
>
> *I am trying to create a new DF , the following statement is not compiling*
>
>
>
> *val* *df** = **sales_demand**.**join**(**product_master**,(*
> *sales_demand**.INVENTORY_ITEM_ID**==**product_master*
> *.INVENTORY_ITEM_ID),**joinType**=**"inner"**)*
>
>
>
>
>
>
>
> *What am I doing wrong?*
>
>
>
> *==Code===*
>
>
>
> *var* sales_order_sql_stmt = s"""SELECT ORDER_NUMBER , INVENTORY_ITEM_ID,
> ORGANIZATION_ID,
>
>   from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd'),
> '-MM-dd') AS schedule_date
>
>   FROM sales_order_demand
>
>   WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'-MM-dd') >= $
> planning_start_date  limit 10"""
>
>
>
> *val* sales_demand = spark.sql (sales_order_sql_stmt)
>
>
>
> //print the data
>
> *sales_demand**.**collect**()*.foreach { println }
>
>
>
>
>
> *val* product_sql_stmt = "select 
> SEGMENT1,INVENTORY_ITEM_ID,ORGANIZATION_ID
> from product limit 10"
>
> *val* product_master = spark.sql (product_sql_stmt)
>
>
>
> //print the data
>
> *product_master**.**collect**()*.foreach { println }
>
>
>
> *val* *df** = **sales_demand**.**join**(**product_master**,(*
> *sales_demand**.INVENTORY_ITEM_ID**==**product_master*
> *.INVENTORY_ITEM_ID),**joinType**=**"inner"**)*
>
>
>
>
>
>
>
>spark.stop()
>