[
https://issues.apache.org/jira/browse/SPARK-46409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-46409:
------------------------------------
Assignee: Vsevolod Stepanov
> Spark Connect Repl does not work with ClosureCleaner
> ----------------------------------------------------
>
> Key: SPARK-46409
> URL: https://issues.apache.org/jira/browse/SPARK-46409
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.0.0
> Reporter: Vsevolod Stepanov
> Assignee: Vsevolod Stepanov
> Priority: Major
> Labels: pull-request-available
>
> SPARK-45136 added ClosureCleaner support to SparkConnect client.
> Unfortunately, this change breaks ConnectRepl launched by
> `./connector/connect/bin/spark-connect-scala-client`. To reproduce the issue:
> # Run `./connector/connect/bin/spark-connect-shell`
> # Run `./connector/connect/bin/spark-connect-scala-client`
> # In the REPL, execute this code:
> ```
> @ def plus1(x: Int): Int = x + 1
> @ val plus1_udf = udf(plus1 _)
> ```
> This will fail with the following error:
> ```
> java.lang.reflect.InaccessibleObjectException: Unable to make private native
> java.lang.reflect.Field[] java.lang.Class.getDeclaredFields0(boolean)
> accessible: module java.base does not "opens java.lang" to unnamed module
> @45099dd3
>
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
>
> java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
> java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
> java.lang.reflect.Method.setAccessible(Method.java:193)
>
> org.apache.spark.util.ClosureCleaner$.getFinalModifiersFieldForJava17(ClosureCleaner.scala:577)
>
> org.apache.spark.util.ClosureCleaner$.setFieldAndIgnoreModifiers(ClosureCleaner.scala:560)
>
> org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18(ClosureCleaner.scala:533)
>
> org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18$adapted(ClosureCleaner.scala:525)
> scala.collection.ArrayOps$WithFilter.foreach(ArrayOps.scala:73)
>
> org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16(ClosureCleaner.scala:525)
>
> org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16$adapted(ClosureCleaner.scala:522)
> scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
> scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
> scala.collection.AbstractIterable.foreach(Iterable.scala:933)
> scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:903)
>
> org.apache.spark.util.ClosureCleaner$.cleanupAmmoniteReplClosure(ClosureCleaner.scala:522)
> org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:251)
>
> org.apache.spark.sql.expressions.SparkConnectClosureCleaner$.clean(UserDefinedFunction.scala:210)
>
> org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:187)
>
> org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:180)
> org.apache.spark.sql.functions$.udf(functions.scala:7956)
> ammonite.$sess.cmd1$Helper.<init>(cmd1.sc:1)
> ammonite.$sess.cmd1$.<clinit>(cmd1.sc:7)
> ```
>
> This is because ClosureCleaner is heavily reliant on using reflection API and
> is not compatible with Java 17. The rest of Spark bypasses this by adding
> `--add-opens` JVM flags, see
> https://issues.apache.org/jira/browse/SPARK-36796. We need to add these
> options to Spark Connect Client launch script as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
