Hi zeppelin-users,
This is my first email to the top-level mailing list. Congratulations for
graduation!
We are hitting some performance issues when multiple users are connected to
the Zeppelin server. From the stack trace, many of the connections are
blocked on a HashMap, which is locked by
org.apache.zeppelin.socket.NotebookServer.broadcastNote.
Our largest notebook is around 800K, and there are around 10 - 20
connections to the Zeppelin server. I think it should be we are
broadcasting some large amount of data to multiple users, and some slow
connections hang the whole web interface.
Is there anyway to reduce the number of broadcasts to improve the web
performance? It is fine for us to refresh and get updates. I've attached
the full stack trace of this issue as well.
Thanks!
Johnny
Blocking Thread:
--
"qtp1874598090-2478" prio=10 tid=0x00007f2fb0003800 nid=0x3373 waiting on
condition [0x00007f329ebe9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000704e15db0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219)
at
org.eclipse.jetty.websocket.common.BlockingWriteCallback$WriteBlocker.block(BlockingWriteCallback.java:83)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:107)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:387)
at
org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:69)
at
org.apache.zeppelin.socket.NotebookServer.broadcast(NotebookServer.java:304)
- locked <0x00000007006b6100> (a java.util.HashMap)
at
org.apache.zeppelin.socket.NotebookServer.broadcastNote(NotebookServer.java:384)
at
org.apache.zeppelin.socket.NotebookServer.updateNote(NotebookServer.java:492)
at
org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:181)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at
org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at
org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Blocked Thread:
--
"qtp1874598090-2498" prio=10 tid=0x00007f306000f800 nid=0x4075 waiting for
monitor entry [0x00007f329eae9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.zeppelin.socket.NotebookServer.addConnectionToNote(NotebookServer.java:229)
- waiting to lock <0x00000007006b6100> (a java.util.HashMap)
at
org.apache.zeppelin.socket.NotebookServer.sendNote(NotebookServer.java:432)
at
org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:145)
at
org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:56)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at
org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at
org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at
org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at
org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at
org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at
org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at
org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
