Vsevolod Stepanov created SPARK-46409:
-----------------------------------------
Summary: Spark Connect Repl does not work with ClosureCleaner
Key: SPARK-46409
URL: https://issues.apache.org/jira/browse/SPARK-46409
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 4.0.0
Reporter: Vsevolod Stepanov
SPARK-45136 added ClosureCleaner support to SparkConnect client. Unfortunately,
this change breaks ConnectRepl launched by
`./connector/connect/bin/spark-connect-scala-client`. To reproduce the issue:
# Run `./connector/connect/bin/spark-connect-shell`
# Run `./connector/connect/bin/spark-connect-scala-client`
# In the REPL, execute this code:
```
@ def plus1(x: Int): Int = x + 1
@ val plus1_udf = udf(plus1 _)
```
This will fail with the following error:
```
java.lang.reflect.InaccessibleObjectException: Unable to make private native
java.lang.reflect.Field[] java.lang.Class.getDeclaredFields0(boolean)
accessible: module java.base does not "opens java.lang" to unnamed module
@45099dd3
java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
java.lang.reflect.Method.setAccessible(Method.java:193)
org.apache.spark.util.ClosureCleaner$.getFinalModifiersFieldForJava17(ClosureCleaner.scala:577)
org.apache.spark.util.ClosureCleaner$.setFieldAndIgnoreModifiers(ClosureCleaner.scala:560)
org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18(ClosureCleaner.scala:533)
org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18$adapted(ClosureCleaner.scala:525)
scala.collection.ArrayOps$WithFilter.foreach(ArrayOps.scala:73)
org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16(ClosureCleaner.scala:525)
org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16$adapted(ClosureCleaner.scala:522)
scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
scala.collection.AbstractIterable.foreach(Iterable.scala:933)
scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:903)
org.apache.spark.util.ClosureCleaner$.cleanupAmmoniteReplClosure(ClosureCleaner.scala:522)
org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:251)
org.apache.spark.sql.expressions.SparkConnectClosureCleaner$.clean(UserDefinedFunction.scala:210)
org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:187)
org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:180)
org.apache.spark.sql.functions$.udf(functions.scala:7956)
ammonite.$sess.cmd1$Helper.<init>(cmd1.sc:1)
ammonite.$sess.cmd1$.<clinit>(cmd1.sc:7)
```
This is because ClosureCleaner is heavily reliant on using reflection API and
is not compatible with Java 17. The rest of Spark bypasses this by adding
`--add-opens` JVM flags, see https://issues.apache.org/jira/browse/SPARK-36796.
We need to add these options to Spark Connect Client launch script as well
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
