[
https://issues.apache.org/jira/browse/SPARK-29033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-29033:
-------------------------------
Description:
Spark 2.x has two separate implementations of the "create named struct"
expression: regular {{CreateNamedStruct}} and {{CreateNamedStructUnsafe}}. The
"unsafe" version was added in SPARK-9373 to support structs in
{{GenerateUnsafeProjection}}. These two expressions both extend the
{{CreateNameStructLike}} trait.
For Spark 3.0, I propose to always use the "unsafe" code path: this will avoid
object allocation / boxing inefficiencies in the "safe" codepath, which is an
especially big problem when generating Encoders for deeply-nested case classes.
was:
Spark 2.x has two separate implementations of the "create named struct"
expression: regular {{CreateNamedStruct}} and {{CreateNamedStructUnsafe}}. The
"unsafe" version was added in SPARK-9373 to support structs in
{{GenerateUnsafeProjection}}. These two expressions both extend the
{{CreateNameStructLike}} trait.
For Spark 3.0, I propose to always use the "unsafe" code path: this will avoid
object allocation / boxing inefficiencies in the "safe" codepath, which is an
especially big problem when generating Encoders for deeply-nested structs.
> Always use CreateNamedStructUnsafe codepath
> -------------------------------------------
>
> Key: SPARK-29033
> URL: https://issues.apache.org/jira/browse/SPARK-29033
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Major
>
> Spark 2.x has two separate implementations of the "create named struct"
> expression: regular {{CreateNamedStruct}} and {{CreateNamedStructUnsafe}}.
> The "unsafe" version was added in SPARK-9373 to support structs in
> {{GenerateUnsafeProjection}}. These two expressions both extend the
> {{CreateNameStructLike}} trait.
> For Spark 3.0, I propose to always use the "unsafe" code path: this will
> avoid object allocation / boxing inefficiencies in the "safe" codepath, which
> is an especially big problem when generating Encoders for deeply-nested case
> classes.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
