kondziolka9ld created SPARK-34583:
-------------------------------------
Summary: typed udf fails when it refers to type member in abstract
class
Key: SPARK-34583
URL: https://issues.apache.org/jira/browse/SPARK-34583
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.1
Reporter: kondziolka9ld
Please consider a following scenario:
{code:java}
scala> abstract class SomeAbstractClass {
| type SomeTypeMember
| }
defined class SomeAbstractClassscala> class SomeSpecificClass extends
SomeAbstractClass {
| override type SomeTypeMember = Int
| }
defined class SomeSpecificClassscala> def someFunction(someInstance:
SomeAbstractClass): Any = {
| udf { _: someInstance.SomeTypeMember => 42 }
| }
someFunction: (someInstance: SomeAbstractClass)Any
scala> someFunction(new SomeSpecificClass)
java.lang.NoClassDefFoundError: no Java class corresponding to
someInstance.SomeTypeMember found
at
scala.reflect.runtime.JavaMirrors$JavaMirror.typeToJavaClass(JavaMirrors.scala:1354)
at
scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:227)
at
scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:68)
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:56)
at org.apache.spark.sql.functions$.$anonfun$udf$1(functions.scala:4509)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.spark.sql.functions$.udf(functions.scala:4509)
at someFunction(<console>:25)
... 47 elided
{code}
On *spark-2.4.7* it works. I guess that it is related to:
{code:java}
In Spark 3.0, using org.apache.spark.sql.functions.udf(AnyRef, DataType) is not
allowed by default. Remove the return type parameter to automatically switch to
typed Scala udf is recommended, or set spark.sql.legacy.allowUntypedScalaUDF to
true to keep using it. In Spark version 2.4 and below, if
org.apache.spark.sql.functions.udf(AnyRef, DataType) gets a Scala closure with
primitive-type argument, the returned UDF returns null if the input values is
null. However, in Spark 3.0, the UDF returns the default value of the Java type
if the input value is null. For example, val f = udf((x: Int) => x,
IntegerType), f($"x") returns null in Spark 2.4 and below if column x is null,
and return 0 in Spark 3.0. This behavior change is introduced because Spark 3.0
is built with Scala 2.12 by default.
{code}
[https://spark.apache.org/docs/latest/sql-migration-guide.html#udfs-and-built-in-functions]
Does spark try to do some type inferation? When it refers to
`SomeAbstractClass.SomeTypeMember` it really does not exist.
Some workaround could be runtime type casting, something like:
{code:java}
udf { param: Any => {
...
param.asInstanceOf[someInstance.SomeTypeMember]
...
}
{code}
----
I classified it as bug since on previous versions of spark it worked. However,
I believe that it can work as designed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
