[ 
https://issues.apache.org/jira/browse/SPARK-60783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145591#comment-18145591
 ] 

Viacheslav Inozemtsev commented on SPARK-60783:
-----------------------------------------------

[~metanil] Are you planning to work on this? If not, I'd like to take it. My 
plan would be to look up invoke and produceResult along the superclass chain, 
so ScalarFunction's default produceResult is still not taken as an 
implementation, with tests for an inherited invoke and an inherited 
produceResult.

> A V2 ScalarFunction that inherits produceResult or invoke from a base class 
> fails with SCALAR_FUNCTION_NOT_FULLY_IMPLEMENTED
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60783
>                 URL: https://issues.apache.org/jira/browse/SPARK-60783
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.3.0
>            Reporter: Anil K
>            Priority: Major
>
> {{V2ExpressionUtils.resolveScalarFunction}} finds a {{ScalarFunction}}'s 
> magic {{invoke}} method or its {{produceResult}} with {{findMethod}}, which 
> calls {{getDeclaredMethod}} on the function's own class. That does not see a 
> method the class inherits from a superclass. So a function that implements 
> {{produceResult}} or {{invoke}} in a base class is reported as not 
> implementing it:
> {code:scala}
> abstract class StrLenBase extends ScalarFunction[Int] {
>   override def produceResult(input: InternalRow): Int = 
> input.getUTF8String(0).numChars()
> }
> class InheritedStrLen extends StrLenBase {
>   override def inputTypes(): Array[DataType] = Array(StringType)
>   override def resultType(): DataType = IntegerType
>   override def name(): String = "strlen_inherited"
> }
> {code}
> Measured on master (0aa4f9136bc), with the function in a catalog:
> * {{SELECT testcat.ns.inherited('abc')}} fails analysis with 
> {{[SCALAR_FUNCTION_NOT_FULLY_IMPLEMENTED] ScalarFunction `strlen_inherited` 
> not implements or overrides method 'produceResult(InternalRow)'}}.
> * The same function with {{produceResult}} declared on the class itself 
> returns 3.
> {{resolveScalarFunction}} is used in three places, so the same function fails 
> in each:
> * a SQL call of the function ({{FunctionResolution}});
> * the write path, which clusters rows by a partition transform 
> ({{DistributionAndOrderingUtils}}, through 
> {{TransformExpression.resolvedFunction}});
> * a storage-partitioned join that evaluates a partition transform (one-side 
> shuffle, identity reduce).
> A fix could look the method up along the superclass chain (e.g. {{getMethod}} 
> for public methods, or walking {{getSuperclass}} for {{getDeclaredMethod}}), 
> stopping before the {{ScalarFunction}} interface, so that the interface's 
> default {{produceResult}} is still not mistaken for an implementation.
> The existing TODO in {{resolveScalarFunction}} covers the opposite case: a 
> Scala class that implements neither method still passes, since Scala adds a 
> forwarder to the interface's default {{produceResult}} that 
> {{getDeclaredMethod}} finds. Walking the superclass chain does not change 
> that.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to