[
https://issues.apache.org/jira/browse/SPARK-47780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-47780:
-----------------------------------
Labels: pull-request-available (was: )
> Generate stable and uniquely-named classes in the Catalyst code generator
> -------------------------------------------------------------------------
>
> Key: SPARK-47780
> URL: https://issues.apache.org/jira/browse/SPARK-47780
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.1
> Reporter: Vojin Jovanovic
> Priority: Minor
> Labels: pull-request-available
>
> Code generated by Catalyst can not be executed on [GraalVM Native
> Image|https://www.graalvm.org/] for the following two reasons:
> # The generated classes for Scala UDFs are not stable so they can not be
> [pre-defined in a native
> image|https://www.graalvm.org/latest/reference-manual/native-image/metadata/ExperimentalAgentOptions/#support-for-predefined-classes].
> The instability comes from the address of hidden functions
> ({{{}/0x<address>{}}}) that [end up in the generated
> code|https://github.com/apache/spark/blob/v3.5.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala#L1173].
> These addresses are completely random and therefore provide little value to
> the end user.
> # The name of the generated class is always the same
> ({{{}org.apache.spark.sql.catalyst.expressions.GeneratedClass{}}}) which
> makes it impossible to [pre-define multiple such classes in a native
> image|https://github.com/oracle/graal/blob/vm-ce-23.1.2/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ClassPredefinitionFeature.java#L153].
> We can fix this by generating the class name in the following format:
> {{{}org.apache.spark.sql.catalyst.expressions.GeneratedClass$<codeHash>{}}}.
> It would be useful enable Spark workloads on GraalVM Native Image. With
> Native Image, one can generate low-memory Spark binaries for their frequent
> workloads. We would also like to make all Spark benchmarks in the
> [renaissance benchmark suite|https://renaissance.dev/] run as native images.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
