Andreas Weise created ZEPPELIN-5598:
---------------------------------------
Summary: DeadLock in RemoteInterpreterServer Close / Cancel
Key: ZEPPELIN-5598
URL: https://issues.apache.org/jira/browse/ZEPPELIN-5598
Project: Zeppelin
Issue Type: Bug
Components: zeppelin-interpreter
Affects Versions: 0.10.0
Reporter: Andreas Weise
Using Interpreter Binding Isolated per User + Scoped per Note, we face dead
lock sitution during Interpreter Shutdown.
Unfortunately we can't provide a full thread dump as Interpreter was running in
container without jstack. Luckily we found Thread Overview of the Driver
Process in the corresponding Spark UI. There are +100 ShutdownThreads which are
all blocked as follows:
{code:java}
15800 ShutdownThread BLOCKED
Blocked by Thread 24
Lock(org.apache.zeppelin.interpreter.InterpreterGroup@2125781891})
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$ShutdownThread.run(RemoteInterpreterServer.java:664)
{code}
{code:java}
Thread 24 BLOCKED
Blocked by Thread 14066
Lock(org.apache.zeppelin.spark.PySparkInterpreter@554374428})
Lock(java.util.concurrent.ThreadPoolExecutor$Worker@188315435}),
Monitor(org.apache.zeppelin.interpreter.InterpreterGroup@2125781891})
org.apache.zeppelin.interpreter.LazyOpenInterpreter.close(LazyOpenInterpreter.java:91)
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.close(RemoteInterpreterServer.java:487)
=> holding
Monitor(org.apache.zeppelin.interpreter.InterpreterGroup@2125781891})
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$close.getResult(RemoteInterpreterService.java:1757)
org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$close.getResult(RemoteInterpreterService.java:1736)
org.apache.zeppelin.shaded.org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
org.apache.zeppelin.shaded.org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
org.apache.zeppelin.shaded.org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
{code}
{code:java}
Thread 14066 Thread-752 BLOCKED
Blocked by Thread 24
Lock(org.apache.zeppelin.interpreter.InterpreterGroup@2125781891})
Monitor(org.apache.zeppelin.spark.PySparkInterpreter@554374428}),
Monitor(org.apache.zeppelin.interpreter.LazyOpenInterpreter@1986146112})
org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:292)
org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:333)
org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:90)
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
=> holding
Monitor(org.apache.zeppelin.interpreter.LazyOpenInterpreter@1986146112})
org.apache.zeppelin.interpreter.LazyOpenInterpreter.cancel(LazyOpenInterpreter.java:118)
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.lambda$cancel$1(RemoteInterpreterServer.java:933)
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$$Lambda$5415/1522577174.run(Unknown
Source)
java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
