Hi Knox Experts,
Needed a help incase you are aware of this issue.
We have this scenario where in we are creating 20 python kernels concurrently, we see that of these around 6-7 kernels fail with "Exception: Remote host closed websocket" because of timeout.
When we tried the same testcase outside of the knox, we did not see the issue. All the kernels got created ok.
So we suspected something in knox which is causing the issue, since the failure is because of the timeout imposed, from the ambari console we changed these 2 parameters “gateway.idle.timeout" and "gateway.websocket.idle.timeout" to different values and ran the testcase.
However we see in the knox logs that although we have set the "gateway.websocket.idle.timeout" to 98 it is being reset to 300000.
We see that idleTimeout is 98 , however it resets and becomes 300000 later on.
So basically it looks like whatever we set is being overwritten to 300000, is this something you would know ?
Here are the 2 lines which shows the values of idleTimeout to 98 and 300000.
2018-04-13 09:17:38,266 DEBUG events.AbstractEventDriver (AbstractEventDriver.java:openSession(223)) - openSession(WebSocketSession[websocket=JettyListenerEventDriver[org.apache.hadoop.gateway.websockets.ProxyWebSocketAdapter],behavior=SERVER,connection=WebSocketServerConnection@3016d0df[IDLE,DecryptedEndPoint@65fc705{/xx.xxx.xxx.xxx:xxxxx<->8443,Open,in,out,-,-,116/98,WebSocketServerConnection}->SelectChannelEndPoint@2dd64f54{/xx.xxx.xxx.xxx:xxxxx<->8443,Open,in,out,-,-,1/98,SslConnection}{io=0,kio=0,kro=1}]{f=Flusher[queueSize=0,aggregateSize=0,failure=null],g=Generator[SERVER,validating],p=Parser@faa724f[ExtensionStack,s=START,c=0,len=0,f=null,p=WebSocketPolicy@430a35ac[behavior=SERVER,maxTextMessageSize=2147483647,maxTextMessageBufferSize=32768,maxBinaryMessageSize=2147483647,maxBinaryMessageBufferSize=32768,asyncWriteTimeout=60000,idleTimeout=98,inputBufferSize=4096]]},remote=WebSocketRemoteEndpoint@63617fae[batching=true],incoming=JettyListenerEventDriver[org.apache.hadoop.gateway.websockets.ProxyWebSocketAdapter],outgoing=ExtensionStack[queueSize=0,extensions=[],incoming=org.eclipse.jetty.websocket.common.WebSocketSession,outgoing=org.eclipse.jetty.websocket.server.WebSocketServerConnection]])
2018-04-13 09:17:40,535 DEBUG component.ContainerLifeCycle (ContainerLifeCycle.java:addBean(324)) - WebSocketSession[websocket=JsrAnnotatedEventDriver[websocket=org.apache.hadoop.gateway.websockets.ProxyInboundSocket@66217bd6],behavior=CLIENT,connection=WebSocketClientConnection@1fcb3472[IDLE,SelectChannelEndPoint@627d9a4b{chs-ksj-013-mn003.bi.services.us-south.bluemix.net/172.16.12.7:8888<->38356,Open,in,out,-,-,1/300000,UpgradeConnection}{io=0,kio=0,kro=1}]{f=Flusher[queueSize=0,aggregateSize=0,failure=null],g=Generator[CLIENT,validating],p=Parser@1eafcb18[ExtensionStack,s=START,c=0,len=0,f=null,p=WebSocketPolicy@4e802544[behavior=CLIENT,maxTextMessageSize=2147483647,maxTextMessageBufferSize=32768,maxBinaryMessageSize=2147483647,maxBinaryMessageBufferSize=32768,asyncWriteTimeout=60000,idleTimeout=300000,inputBufferSize=4096]]},remote=null,incoming=JsrAnnotatedEventDriver[websocket=org.apache.hadoop.gateway.websockets.ProxyInboundSocket@66217bd6],outgoing=ExtensionStack[queueSize=0,extensions=[],incoming=org.eclipse.jetty.websocket.jsr356.JsrSession,outgoing=org.eclipse.jetty.websocket.client.io.WebSocketClientConnection]] added {ExtensionStack[queueSize=0,extensions=[],incoming=org.eclipse.jetty.websocket.jsr356.JsrSession,outgoing=org.eclipse.jetty.websocket.client.io.WebSocketClientConnection],AUTO}
This are the versions : Apache Knox: 0.12.0.2.6.2.0-205 (c8ad94e5bd29a6925f5a667660a50cfc6de69cc3)
HDP-2.6.2.0-205
HDP-2.6.2.0-205
Please let me know if anything more would help
Thanks & Regards,
Ashok Jose
**********************************************************
BigInsights Quality
EGL C Block, 6th Floor
IBM India Software Labs,
Mail-Id : [email protected]
Ashok Jose
**********************************************************
BigInsights Quality
EGL C Block, 6th Floor
IBM India Software Labs,
Mail-Id : [email protected]
