Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22732 For Scala 2.11, we should not introduce any behavior change and also keep binary and source compatibility. ``` scala> import org.apache.spark.sql.types.DataTypes import org.apache.spark.sql.types.DataTypes scala> val f2 = udf({(x: Int) => x}, DataTypes.IntegerType) f2: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(<function1>,IntegerType,None) scala> spark.range(3).select(f2('id + null)).show() +----------------+ |UDF((id + null))| +----------------+ | null| | null| | null| +----------------+ ``` For Scala 2.12, since we are unable to know the type nullability in a few APIs, we issue a warning message in these cases. Below is the example which will generate a different answer: ``` scala> import org.apache.spark.sql.types.DataTypes import org.apache.spark.sql.types.DataTypes scala> val f2 = udf({(x: Int) => x}, DataTypes.IntegerType) f2: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction($Lambda$2801/26868055@5eb35a26,IntegerType,None) scala> spark.range(3).select(f2('id + null)).show() 18/10/18 23:07:02 WARN ScalaReflection: Scala version 2.12.7 cannot get type nullability correctly via reflection, thus Spark cannot add proper input null check for UDF. To avoid this problem, use the typed UDF interfaces instead. +----------------+ |UDF((id + null))| +----------------+ | 0| | 0| | 0| +----------------+ ```
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org