Anil K created SPARK-60783:
------------------------------
Summary: A V2 ScalarFunction that inherits produceResult or invoke
from a base class fails with SCALAR_FUNCTION_NOT_FULLY_IMPLEMENTED
Key: SPARK-60783
URL: https://issues.apache.org/jira/browse/SPARK-60783
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Anil K
{{V2ExpressionUtils.resolveScalarFunction}} finds a {{ScalarFunction}}'s magic
{{invoke}} method or its {{produceResult}} with {{findMethod}}, which calls
{{getDeclaredMethod}} on the function's own class. That does not see a method
the class inherits from a superclass. So a function that implements
{{produceResult}} or {{invoke}} in a base class is reported as not implementing
it:
{code:scala}
abstract class StrLenBase extends ScalarFunction[Int] {
override def produceResult(input: InternalRow): Int =
input.getUTF8String(0).numChars()
}
class InheritedStrLen extends StrLenBase {
override def inputTypes(): Array[DataType] = Array(StringType)
override def resultType(): DataType = IntegerType
override def name(): String = "strlen_inherited"
}
{code}
Measured on master (0aa4f9136bc), with the function in a catalog:
* {{SELECT testcat.ns.inherited('abc')}} fails analysis with
{{[SCALAR_FUNCTION_NOT_FULLY_IMPLEMENTED] ScalarFunction `strlen_inherited` not
implements or overrides method 'produceResult(InternalRow)'}}.
* The same function with {{produceResult}} declared on the class itself returns
3.
{{resolveScalarFunction}} is used in three places, so the same function fails
in each:
* a SQL call of the function ({{FunctionResolution}});
* the write path, which clusters rows by a partition transform
({{DistributionAndOrderingUtils}}, through
{{TransformExpression.resolvedFunction}});
* a storage-partitioned join that evaluates a partition transform (one-side
shuffle, identity reduce).
A fix could look the method up along the superclass chain (e.g. {{getMethod}}
for public methods, or walking {{getSuperclass}} for {{getDeclaredMethod}}),
stopping before the {{ScalarFunction}} interface, so that the interface's
default {{produceResult}} is still not mistaken for an implementation.
The existing TODO in {{resolveScalarFunction}} covers the opposite case: a
Scala class that implements neither method still passes, since Scala adds a
forwarder to the interface's default {{produceResult}} that
{{getDeclaredMethod}} finds. Walking the superclass chain does not change that.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
