[
https://issues.apache.org/jira/browse/SPARK-60811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Pan updated SPARK-60811:
------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Generated code cannot resolve classes from jars added at runtime when the
> driver thread's context classloader does not include them
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60811
> URL: https://issues.apache.org/jira/browse/SPARK-60811
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Cheng Pan
> Priority: Major
>
> Spark compiles generated code with the context classloader of the driver
> thread that runs the query. Classes from jars added at runtime ({{ADD JAR}},
> {{CREATE FUNCTION ... USING JAR}}) are only visible through it when the jar
> loading switched that thread's context classloader. When it did not, or the
> classloader was reset (for example, on a later line in spark-shell, on
> another driver thread, or in a Spark Connect session without
> {{spark.sql.artifact.isolation.alwaysApplyClassloader=true}}), compiling
> fails with {{Cannot determine simple type name}}. Whole-stage codegen falls
> back to interpreted execution by default, but operators without a fallback,
> such as {{ORDER BY f(c) LIMIT 10}}, fail the query.
> One way to hit this is a DataSource V2 function's magic {{invoke}} method,
> whose class is referenced by name in generated code.
> Repro (spark-shell):
> {code}
> scala> sql("CREATE TEMPORARY FUNCTION f AS 'com.example.StrLen' USING JAR
> '/tmp/strlen.jar'")
> scala> sql("SELECT f(c) FROM t ORDER BY f(c) LIMIT 10").show() // fails
> {code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
