[
https://issues.apache.org/jira/browse/SPARK-56448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Zsolt Piros resolved SPARK-56448.
----------------------------------------
Fix Version/s: 4.2.0
3.5.9
4.1.2
4.0.3
Resolution: Fixed
Issue resolved by pull request 55720
[https://github.com/apache/spark/pull/55720]
> Spark connect client fails on subsequent start due to ammonite (code) cache
> ---------------------------------------------------------------------------
>
> Key: SPARK-56448
> URL: https://issues.apache.org/jira/browse/SPARK-56448
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.2.0
> Reporter: Attila Zsolt Piros
> Assignee: Anupam Yadav
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.0, 3.5.9, 4.1.2, 4.0.3
>
>
> When Spark Connect client is restarted (from the same working directory) it
> causes the following:
> {noformat}
> ➜ spark git:(SPARK-connect) ✗ bin/spark-shell --remote
> "sc://localhost:15002"
> WARNING: Using incubator modules: jdk.incubator.vector
> 26/04/10 16:04:35 INFO BaseAllocator: Debug mode disabled. Enable with the VM
> option -Darrow.memory.debug.allocator=true.
> 26/04/10 16:04:35 INFO DefaultAllocationManagerOption: allocation manager
> type not specified, using netty as the default type
> 26/04/10 16:04:35 INFO CheckAllocator: Using DefaultAllocationManager at
> memory/netty/DefaultAllocationManagerFactory.class
> Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
> Compiling /Users/apiros/git/attilapiros/spark/(console)
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 4.2.0-SNAPSHOT
> /_/
> Type in expressions to have them evaluated.
> Spark connect server version 4.2.0-SNAPSHOT.
> Spark session available as 'spark'.
> scala> Bye!
> ➜ spark git:(SPARK-connect) ✗ bin/spark-shell --remote
> "sc://localhost:15002"
> WARNING: Using incubator modules: jdk.incubator.vector
> 26/04/10 16:04:42 INFO BaseAllocator: Debug mode disabled. Enable with the VM
> option -Darrow.memory.debug.allocator=true.
> 26/04/10 16:04:42 INFO DefaultAllocationManagerOption: allocation manager
> type not specified, using netty as the default type
> 26/04/10 16:04:42 INFO CheckAllocator: Using DefaultAllocationManager at
> memory/netty/DefaultAllocationManagerFactory.class
> Exception in thread "main" java.lang.NullPointerException: Cannot invoke
> "ammonite.predef.ArgsPredef$Helper.spark()" because the return value of
> "ammonite.predef.CodePredef.ArgsPredef()" is null
> at ammonite.predef.CodePredef$Helper.<init>((console):7)
> at ammonite.predef.CodePredef$.<clinit>((console):6)
> at ammonite.predef.CodePredef.$main((console))
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:569)
> at
> ammonite.runtime.Evaluator$$anon$1.$anonfun$evalMain$1(Evaluator.scala:108)
> at ammonite.util.Util$.withContextClassloader(Util.scala:21)
> at ammonite.runtime.Evaluator$$anon$1.evalMain(Evaluator.scala:90)
> at
> ammonite.interp.Interpreter.$anonfun$processAllScriptBlocks$10(Interpreter.scala:594)
> at ammonite.util.Res$Success.map(Res.scala:63)
> at
> ammonite.interp.Interpreter.$anonfun$processAllScriptBlocks$9(Interpreter.scala:594)
> at scala.Option$WithFilter.map(Option.scala:242)
> at ammonite.interp.Interpreter.loop$1(Interpreter.scala:574)
> at
> ammonite.interp.Interpreter.processAllScriptBlocks(Interpreter.scala:644)
> at
> ammonite.interp.Interpreter.$anonfun$processModule$6(Interpreter.scala:432)
> at ammonite.util.Catching.flatMap(Res.scala:110)
> at
> ammonite.interp.Interpreter.$anonfun$processModule$5(Interpreter.scala:423)
> at ammonite.util.Res$Success.flatMap(Res.scala:58)
> at ammonite.interp.Interpreter.processModule(Interpreter.scala:413)
> at
> ammonite.interp.Interpreter.$anonfun$initializePredef$3(Interpreter.scala:142)
> at
> ammonite.interp.Interpreter.$anonfun$initializePredef$3$adapted(Interpreter.scala:142)
> at
> ammonite.interp.PredefInitialization$.$anonfun$apply$2(PredefInitialization.scala:79)
> at ammonite.util.Res$.$anonfun$fold$1(Res.scala:27)
> at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
> at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
> at scala.collection.immutable.List.foldLeft(List.scala:79)
> at ammonite.util.Res$.fold(Res.scala:25)
> at
> ammonite.interp.PredefInitialization$.apply(PredefInitialization.scala:67)
> at ammonite.interp.Interpreter.initializePredef(Interpreter.scala:144)
> at ammonite.repl.Repl.initializePredef(Repl.scala:150)
> at ammonite.Main.run(Main.scala:231)
> at
> org.apache.spark.sql.application.ConnectRepl$.$anonfun$doMain$1(ConnectRepl.scala:168)
> {noformat}
> This is due to the ~/.ammonite directory persisting pre-defined code caches
> (predef).
> So when I start it from a clean directory it works just fine again or when
> when I delete the "~/.ammonite" directory:
> {noformat}
> ➜ spark git:(SPARK-connect) ✗ rm -rf ~/.ammonite
> ➜ spark git:(SPARK-connect) ✗ bin/spark-shell --remote
> "sc://localhost:15002"
> WARNING: Using incubator modules: jdk.incubator.vector
> 26/04/10 16:36:31 INFO BaseAllocator: Debug mode disabled. Enable with the VM
> option -Darrow.memory.debug.allocator=true.
> 26/04/10 16:36:31 INFO DefaultAllocationManagerOption: allocation manager
> type not specified, using netty as the default type
> 26/04/10 16:36:31 INFO CheckAllocator: Using DefaultAllocationManager at
> memory/netty/DefaultAllocationManagerFactory.class
> Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
> Compiling /Users/apiros/git/attilapiros/spark/(console)
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 4.2.0-SNAPSHOT
> /_/
> Type in expressions to have them evaluated.
> Spark connect server version 4.2.0-SNAPSHOT.
> Spark session available as 'spark'.
> scala>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
