[ https://issues.apache.org/jira/browse/SPARK-46409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-46409. ---------------------------------- Fix Version/s: 4.0.0 Resolution: Fixed Issue resolved by pull request 44360 [https://github.com/apache/spark/pull/44360] > Spark Connect Repl does not work with ClosureCleaner > ---------------------------------------------------- > > Key: SPARK-46409 > URL: https://issues.apache.org/jira/browse/SPARK-46409 > Project: Spark > Issue Type: Bug > Components: Connect > Affects Versions: 4.0.0 > Reporter: Vsevolod Stepanov > Assignee: Vsevolod Stepanov > Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > SPARK-45136 added ClosureCleaner support to SparkConnect client. > Unfortunately, this change breaks ConnectRepl launched by > `./connector/connect/bin/spark-connect-scala-client`. To reproduce the issue: > # Run `./connector/connect/bin/spark-connect-shell` > # Run `./connector/connect/bin/spark-connect-scala-client` > # In the REPL, execute this code: > ``` > @ def plus1(x: Int): Int = x + 1 > @ val plus1_udf = udf(plus1 _) > ``` > This will fail with the following error: > ``` > java.lang.reflect.InaccessibleObjectException: Unable to make private native > java.lang.reflect.Field[] java.lang.Class.getDeclaredFields0(boolean) > accessible: module java.base does not "opens java.lang" to unnamed module > @45099dd3 > > java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354) > > java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297) > java.lang.reflect.Method.checkCanSetAccessible(Method.java:199) > java.lang.reflect.Method.setAccessible(Method.java:193) > > org.apache.spark.util.ClosureCleaner$.getFinalModifiersFieldForJava17(ClosureCleaner.scala:577) > > org.apache.spark.util.ClosureCleaner$.setFieldAndIgnoreModifiers(ClosureCleaner.scala:560) > > org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18(ClosureCleaner.scala:533) > > org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$18$adapted(ClosureCleaner.scala:525) > scala.collection.ArrayOps$WithFilter.foreach(ArrayOps.scala:73) > > org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16(ClosureCleaner.scala:525) > > org.apache.spark.util.ClosureCleaner$.$anonfun$cleanupAmmoniteReplClosure$16$adapted(ClosureCleaner.scala:522) > scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576) > scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574) > scala.collection.AbstractIterable.foreach(Iterable.scala:933) > scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:903) > > org.apache.spark.util.ClosureCleaner$.cleanupAmmoniteReplClosure(ClosureCleaner.scala:522) > org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:251) > > org.apache.spark.sql.expressions.SparkConnectClosureCleaner$.clean(UserDefinedFunction.scala:210) > > org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:187) > > org.apache.spark.sql.expressions.ScalarUserDefinedFunction$.apply(UserDefinedFunction.scala:180) > org.apache.spark.sql.functions$.udf(functions.scala:7956) > ammonite.$sess.cmd1$Helper.<init>(cmd1.sc:1) > ammonite.$sess.cmd1$.<clinit>(cmd1.sc:7) > ``` > > This is because ClosureCleaner is heavily reliant on using reflection API and > is not compatible with Java 17. The rest of Spark bypasses this by adding > `--add-opens` JVM flags, see > https://issues.apache.org/jira/browse/SPARK-36796. We need to add these > options to Spark Connect Client launch script as well -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org