[
https://issues.apache.org/jira/browse/SPARK-61774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61774:
-----------------------------------
Labels: pull-request-available (was: )
> Add columnar scalar UDFs, evaluated by an operator that native engines can
> replace
> ----------------------------------------------------------------------------------
>
> Key: SPARK-61774
> URL: https://issues.apache.org/jira/browse/SPARK-61774
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> Spark's scalar UDFs are evaluated row by row. Over a columnar source this
> costs a conversion per row, most visibly for strings, where each value is
> decoded to a Java String and encoded back. It also leaves native engines that
> replace Spark's operators without a function boundary they can take over: an
> engine either runs the row-based function itself or defines a private
> mechanism that plain Spark cannot run.
> This proposes a columnar scalar UDF:
> * {{ColumnarScalarFunction}}, a JVM interface that takes the arguments as a
> {{ColumnarBatch}} and returns a {{ColumnVector}}.
> * A function definition that holds an optional JVM implementation and a list
> of opaque engine implementations (a kind and properties), which Spark does
> not interpret.
> * A logical and a physical operator, {{EvalColumnarUDFs}}, that the calls are
> extracted into, as Python UDFs are. The physical operator is columnar, passes
> its input columns through and appends one column per call.
> Spark runs the JVM implementation. An engine that replaces the physical
> operator can run its own implementation of the same function instead, and the
> query falls back to the JVM implementation when it does not. A function with
> no JVM implementation is allowed and fails at execution if no engine has
> taken the operator.
> The operator targets columnar input. Over a row-based child, Spark inserts
> its usual row to columnar transition.
> In a benchmark over Parquet, the JVM columnar implementation is 1.2x to 2.6x
> faster than the equivalent Scala UDF, with the larger gains on string
> functions.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]