Hi,
I have a UDF created and registered as below
val df =
sqlContext.read.format("com.databricks.spark.csv").option("inferSchema",
"true").option("header", "true").load("/data/stg/table2")
val current_date = sqlContext.sql("SELECT FROM_unixtime(unix_timestamp(),
'dd/MM/yyyy') ").collect.apply(0).getString(0)
def ChangeDate(word : String) : String = {
return
word.substring(6,10)+"-"+word.substring(3,5)+"-"+word.substring(0,2)
}
sqlContext.udf.register("ChangeDate", ChangeDate(_:String))
Now I want to filter on all Payment date older than 6 months. I do this
scala> df.filter(months_between(lit(current_date),ChangeDate(col("Payment
date")).toDate) > 6)
<console>:29: error: type mismatch;
found : org.apache.spark.sql.Column
required: String
df.filter(months_between(lit(current_date),ChangeDate(col("Payment
date")).toDate) > 6)
It comes back with the above error. Any ideas?
Thanks
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
