Hello I'm having problems with UDF, I was reading a bit about it , and it look's like a closure issue, but I don't know hoy to fix it, it works fine on 2.11.
my code for udf definition (I tried several posibilites this is the las one) val o:org.apache.spark.sql.api.java.UDF2[java.sql.Timestamp,String, Int] = (date:java.sql.Timestamp, periodExp:String) => { 11 } val o2:org.apache.spark.sql.api.java.UDF2[java.sql.Date,String, Int] = (date:java.sql.Date, periodExp:String) => { 11 } sc.udf.register ("timePeriod",o2, DataTypes.IntegerType ) sc.udf.register("timePeriod", o , DataTypes.IntegerType ) exception Caused by: java.io.NotSerializableException: scala.runtime.LazyRef [info] Serialization stack: [info] - object not serializable (class: scala.runtime.LazyRef, value: LazyRef thunk) [info] - element of array (index: 3) [info] - array (class [Ljava.lang.Object;, size 5) [info] - field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;) [info] - object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=class org.apache.spark.sql.catalyst.expressions.ScalaUDF, functionalInterfaceMethod=scala/Function1.apply:(Ljava/lang/Object;)Ljava/lang/Object;, implementation=invokeStatic org/apache/spark/sql/catalyst/expressions/ScalaUDF.$anonfun$f$3:(Lscala/Function2;Lorg/apache/spark/sql/catalyst/expressions/Expression;Lorg/apache/spark/sql/catalyst/expressions/Expression;Lscala/runtime/LazyRef;Lscala/runtime/LazyRef;Lorg/apache/spark/sql/catalyst/InternalRow;)Ljava/lang/Object;, instantiatedMethodType=(Lorg/apache/spark/sql/catalyst/InternalRow;)Ljava/lang/Object;, numCaptured=5]) Regards, -- Ing. Ivaldi Andres