Spark 2.0 - Join statement compile error

[Subhajit Purkayastha]

All,

 

I have the following dataFrames and the temp table. 

 

I am trying to create a new DF , the following statement is not compiling

 

val df =
sales_demand.join(product_master,(sales_demand.INVENTORY_ITEM_ID==product_ma
ster.INVENTORY_ITEM_ID),joinType="inner")

    

 



 

What am I doing wrong?

 

==Code===

 

var sales_order_sql_stmt = s"""SELECT ORDER_NUMBER , INVENTORY_ITEM_ID,
ORGANIZATION_ID,

          from_unixtime(unix_timestamp(SCHEDULE_SHIP_DATE,'yyyy-MM-dd'),
'yyyy-MM-dd') AS schedule_date

      FROM sales_order_demand 

      WHERE unix_timestamp(SCHEDULE_SHIP_DATE,'yyyy-MM-dd') >=
$planning_start_date  limit 10"""

      

    val sales_demand = spark.sql (sales_order_sql_stmt)

    

    //print the data

    sales_demand.collect().foreach { println }

    

    

    val product_sql_stmt = "select
SEGMENT1,INVENTORY_ITEM_ID,ORGANIZATION_ID from product limit 10"    

    val product_master = spark.sql (product_sql_stmt)

    

    //print the data

    product_master.collect().foreach { println }

  

    val df =
sales_demand.join(product_master,(sales_demand.INVENTORY_ITEM_ID==product_ma
ster.INVENTORY_ITEM_ID),joinType="inner")

    

 

 

   spark.stop()