[ 
https://issues.apache.org/jira/browse/SPARK-61774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61774:
-----------------------------------
    Labels: pull-request-available  (was: )

> Add columnar scalar UDFs, evaluated by an operator that native engines can 
> replace
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-61774
>                 URL: https://issues.apache.org/jira/browse/SPARK-61774
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>
> Spark's scalar UDFs are evaluated row by row. Over a columnar source this 
> costs a conversion per row, most visibly for strings, where each value is 
> decoded to a Java String and encoded back. It also leaves native engines that 
> replace Spark's operators without a function boundary they can take over: an 
> engine either runs the row-based function itself or defines a private 
> mechanism that plain Spark cannot run.
> This proposes a columnar scalar UDF:
> * {{ColumnarScalarFunction}}, a JVM interface that takes the arguments as a 
> {{ColumnarBatch}} and returns a {{ColumnVector}}.
> * A function definition that holds an optional JVM implementation and a list 
> of opaque engine implementations (a kind and properties), which Spark does 
> not interpret.
> * A logical and a physical operator, {{EvalColumnarUDFs}}, that the calls are 
> extracted into, as Python UDFs are. The physical operator is columnar, passes 
> its input columns through and appends one column per call.
> Spark runs the JVM implementation. An engine that replaces the physical 
> operator can run its own implementation of the same function instead, and the 
> query falls back to the JVM implementation when it does not. A function with 
> no JVM implementation is allowed and fails at execution if no engine has 
> taken the operator.
> The operator targets columnar input. Over a row-based child, Spark inserts 
> its usual row to columnar transition.
> In a benchmark over Parquet, the JVM columnar implementation is 1.2x to 2.6x 
> faster than the equivalent Scala UDF, with the larger gains on string 
> functions.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to