Vojin Jovanovic created SPARK-47780:
---------------------------------------
Summary: Generate stable and uniquely-named classes in the
Catalyst code generator
Key: SPARK-47780
URL: https://issues.apache.org/jira/browse/SPARK-47780
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.5.1
Reporter: Vojin Jovanovic
Code generated by Catalyst can not be executed on [GraalVM Native
Image|https://www.graalvm.org/] for the following two reasons:
# The generated classes for Scala UDFs are not stable so they can not be
[pre-defined in a native
image|https://www.graalvm.org/latest/reference-manual/native-image/metadata/ExperimentalAgentOptions/#support-for-predefined-classes].
The instability comes from the address of hidden functions
({{{}/0x<address>{}}}) that [end up in the generated
code|https://github.com/apache/spark/blob/v3.5.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala#L1173].
These addresses are completely random and therefore provide little value to
the end user.
# The name of the generated class is always the same
({{{}org.apache.spark.sql.catalyst.expressions.GeneratedClass{}}}) which makes
it impossible to [pre-define multiple such classes in a native
image|https://github.com/oracle/graal/blob/vm-ce-23.1.2/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ClassPredefinitionFeature.java#L153].
We can fix this by generating the class name in the following format:
{{{}org.apache.spark.sql.catalyst.expressions.GeneratedClass$<codeHash>{}}}.
It would be useful enable Spark workloads on GraalVM Native Image. With Native
Image, one can generate low-memory Spark binaries for their frequent workloads.
We would also like to make all Spark benchmarks in the [renaissance benchmark
suite|https://renaissance.dev/] run as native images.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
