> -----Original Message----- > From: Mark Thomas [mailto:ma...@apache.org] > Sent: Wednesday, November 27, 2013 12:59 PM > To: Tomcat Users List > Subject: Re: seeing RemoteEndpoint.Basic BINARY_FULL_WRITING errors under > load in Tomcat 7.0.48-dev (trunk) > > On 27/11/2013 17:28, Bob DeRemer wrote: > > When testing our WebSocket application under heavy load we eventually get > the following exception from the JSR-356 ServerEndpoint's > RemoteEndpoint.Basic when trying to write our binary response message: > > > > java.lang.IllegalStateException: The remote endpoint was in state > [BINARY_FULL_WRITING] which is an invalid state for called method > > at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.chec > kState(WsRemoteEndpointImplBase.java:1048) ~[tomcat7- > websocket.jar:7.0.48-dev] > > at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase$StateMachine.bina > ryStart(WsRemoteEndpointImplBase.java:1001) ~[tomcat7- > websocket.jar:7.0.48-dev] > > at > org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendBytes(WsRemo > teEndpointImplBase.java:118) ~[tomcat7-websocket.jar:7.0.48-dev] > > at > org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendBinary(WsRemote > EndpointBasic.java:43) ~[tomcat7-websocket.jar:7.0.48-dev] > > at > com.thingworx.communications.server.connection.Jsr356ServerConnection.sen > dBinaryMessage(Jsr356ServerConnection.java:240) ~[thingworx-server- > common.jar:na] > > at > com.thingworx.communications.common.endpoints.CommunicationEndpoint.se > ndBinaryMessage(CommunicationEndpoint.java:321) ~[thingworx-server- > common.jar:na] > > at > com.thingworx.communications.common.endpoints.CommunicationEndpoint.se > ndMessage(CommunicationEndpoint.java:303) ~[thingworx-server- > common.jar:na] > > at > com.thingworx.apiserver.endpoints.RoutingServerEndpoint.sendMessage(Routi > ngServerEndpoint.java:24) ~[thingworx-apiserver.jar:na] > > at > com.thingworx.apiserver.routing.OutboundMessageRoutingTask.run(Outbound > MessageRoutingTask.java:79) ~[thingworx-apiserver.jar:na] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 > 145) [na:1.7.0_45] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: > 615) [na:1.7.0_45] > > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > > > > The Tomcat build is 7.0.48-dev (from today, 11/27) and we're running on > Ubuntu Server 13.10 in EC2 (16 vCPUs). The client created 1000 concurrent > websockets, and is using 100 threads to send 100 concurrent requests every 3 > ms. The server receives the messages, processes them, then sends the > response back to the appropriate client. Many response are sent successfully, > but after some period of time, we start receiving the above errors when > attempting to send the responses back. > > > > Is it possible that we could be filling up the TCP buffers (or something > > related) > and this would cause the above exception? > > What can cause the above exception is if the previous message on that > endpoint failed with an IOException. The nature of WebSocket is such that once > this happens the only option is to close the connection and start a new one. >
It appears that we are getting the following IOException: java.io.IOException: java.util.concurrent.TimeoutException immediately before we got the BINARY_FULL_WRITING errors. Would this TimeoutException be caused because the underlying socket did not receive an ACK for the sent websocket message, or could something else cause this when trying to write to the websocket? Thx - bob > > Please note: we lock around a specific ServerEndpoint's > RemoteEndpoint.Basic - to make sure we aren't trying to send more than 1 > message at a time. > > > > Please let me know if this is a bug, or if you need additional information. > > A bug is always possible but at this point it looks like some error handling > is > missing in the app. > > Mark > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
