Attila Zsolt Piros created SPARK-56448:
------------------------------------------
Summary: Spark connect client fails on subsequent start due to
ammonite (code) cache
Key: SPARK-56448
URL: https://issues.apache.org/jira/browse/SPARK-56448
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 4.2.0
Reporter: Attila Zsolt Piros
When Spark Connect client is restarted (from the same working directory) it
causes the following:
{noformat}
➜ spark git:(SPARK-connect) ✗ bin/spark-shell --remote "sc://localhost:15002"
WARNING: Using incubator modules: jdk.incubator.vector
26/04/10 16:04:35 INFO BaseAllocator: Debug mode disabled. Enable with the VM
option -Darrow.memory.debug.allocator=true.
26/04/10 16:04:35 INFO DefaultAllocationManagerOption: allocation manager type
not specified, using netty as the default type
26/04/10 16:04:35 INFO CheckAllocator: Using DefaultAllocationManager at
memory/netty/DefaultAllocationManagerFactory.class
Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
Compiling /Users/apiros/git/attilapiros/spark/(console)
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.2.0-SNAPSHOT
/_/
Type in expressions to have them evaluated.
Spark connect server version 4.2.0-SNAPSHOT.
Spark session available as 'spark'.
scala> Bye!
➜ spark git:(SPARK-connect) ✗ bin/spark-shell --remote "sc://localhost:15002"
WARNING: Using incubator modules: jdk.incubator.vector
26/04/10 16:04:42 INFO BaseAllocator: Debug mode disabled. Enable with the VM
option -Darrow.memory.debug.allocator=true.
26/04/10 16:04:42 INFO DefaultAllocationManagerOption: allocation manager type
not specified, using netty as the default type
26/04/10 16:04:42 INFO CheckAllocator: Using DefaultAllocationManager at
memory/netty/DefaultAllocationManagerFactory.class
Exception in thread "main" java.lang.NullPointerException: Cannot invoke
"ammonite.predef.ArgsPredef$Helper.spark()" because the return value of
"ammonite.predef.CodePredef.ArgsPredef()" is null
at ammonite.predef.CodePredef$Helper.<init>((console):7)
at ammonite.predef.CodePredef$.<clinit>((console):6)
at ammonite.predef.CodePredef.$main((console))
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at
ammonite.runtime.Evaluator$$anon$1.$anonfun$evalMain$1(Evaluator.scala:108)
at ammonite.util.Util$.withContextClassloader(Util.scala:21)
at ammonite.runtime.Evaluator$$anon$1.evalMain(Evaluator.scala:90)
at
ammonite.interp.Interpreter.$anonfun$processAllScriptBlocks$10(Interpreter.scala:594)
at ammonite.util.Res$Success.map(Res.scala:63)
at
ammonite.interp.Interpreter.$anonfun$processAllScriptBlocks$9(Interpreter.scala:594)
at scala.Option$WithFilter.map(Option.scala:242)
at ammonite.interp.Interpreter.loop$1(Interpreter.scala:574)
at
ammonite.interp.Interpreter.processAllScriptBlocks(Interpreter.scala:644)
at
ammonite.interp.Interpreter.$anonfun$processModule$6(Interpreter.scala:432)
at ammonite.util.Catching.flatMap(Res.scala:110)
at
ammonite.interp.Interpreter.$anonfun$processModule$5(Interpreter.scala:423)
at ammonite.util.Res$Success.flatMap(Res.scala:58)
at ammonite.interp.Interpreter.processModule(Interpreter.scala:413)
at
ammonite.interp.Interpreter.$anonfun$initializePredef$3(Interpreter.scala:142)
at
ammonite.interp.Interpreter.$anonfun$initializePredef$3$adapted(Interpreter.scala:142)
at
ammonite.interp.PredefInitialization$.$anonfun$apply$2(PredefInitialization.scala:79)
at ammonite.util.Res$.$anonfun$fold$1(Res.scala:27)
at scala.collection.LinearSeqOps.foldLeft(LinearSeq.scala:183)
at scala.collection.LinearSeqOps.foldLeft$(LinearSeq.scala:179)
at scala.collection.immutable.List.foldLeft(List.scala:79)
at ammonite.util.Res$.fold(Res.scala:25)
at
ammonite.interp.PredefInitialization$.apply(PredefInitialization.scala:67)
at ammonite.interp.Interpreter.initializePredef(Interpreter.scala:144)
at ammonite.repl.Repl.initializePredef(Repl.scala:150)
at ammonite.Main.run(Main.scala:231)
at
org.apache.spark.sql.application.ConnectRepl$.$anonfun$doMain$1(ConnectRepl.scala:168)
{noformat}
This is due to the ~/.ammonite directory persisting pre-defined code caches
(predef).
So when I start it from a clean directory it works just fine again or when when
I delete the "~/.ammonite" directory:
{noformat}
➜ spark git:(SPARK-connect) ✗ rm -rf ~/.ammonite
➜ spark git:(SPARK-connect) ✗ bin/spark-shell --remote "sc://localhost:15002"
WARNING: Using incubator modules: jdk.incubator.vector
26/04/10 16:36:31 INFO BaseAllocator: Debug mode disabled. Enable with the VM
option -Darrow.memory.debug.allocator=true.
26/04/10 16:36:31 INFO DefaultAllocationManagerOption: allocation manager type
not specified, using netty as the default type
26/04/10 16:36:31 INFO CheckAllocator: Using DefaultAllocationManager at
memory/netty/DefaultAllocationManagerFactory.class
Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
Compiling /Users/apiros/git/attilapiros/spark/(console)
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 4.2.0-SNAPSHOT
/_/
Type in expressions to have them evaluated.
Spark connect server version 4.2.0-SNAPSHOT.
Spark session available as 'spark'.
scala>
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
