The file descriptor peak show up in our monitoring application. We have some charts showing the number of file descriptors owned by the tomcat process (ls /proc/$(pgrep -u tomcat7)/fd/ | wc -l).
The calatalina.out log shows errors, the most frequent being a java.io.IOException: Broken pipe. Apr 20, 2015 12:11:02 AM org.apache.coyote.AbstractProcessor setErrorState INFO: An error occurred in processing while on a non-container thread. The connection will be closed immediately java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:128) at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101) at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:174) at org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:163) at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:242) at org.apache.coyote.http11.InternalNioOutputBuffer.flush(InternalNioOutputBuffer.java:94) at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:801) at org.apache.coyote.Response.action(Response.java:172) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:363) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331) at org.apache.catalina.connector.CoyoteWriter.flush(CoyoteWriter.java:98) at networkComm.commands.HttpCommand.sendResponse(HttpCommand.java:223) at com.stickyadstv.adex.AuctioneerResponseWriter.respondToClient(AuctioneerResponseWriter.java:322) at com.stickyadstv.adex.BidSerializationListener.checkSerializationIsComplete(BidSerializationListener.java:70) at com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:53) at com.stickyadstv.adex.BidSerializationListener.completed(BidSerializationListener.java:24) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115) at com.stickyadstv.adex.bidder.openrtb.OpenRTBBid.serializeAdm(OpenRTBBid.java:158) at com.stickyadstv.adex.bidder.openrtb.OpenRTBBid$AdmReceived.completed(OpenRTBBid.java:310) at com.stickyadstv.adex.bidder.openrtb.OpenRTBBid$AdmReceived.completed(OpenRTBBid.java:254) at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:115) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:173) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:355) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:242) at org.apache.http.impl.nio.client.LoggingAsyncRequestExecutor.inputReady(LoggingAsyncRequestExecutor.java:87) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:264) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:73) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:37) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:113) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:584) at java.lang.Thread.run(Thread.java:745) Here a count made on "Exception": 304 java.io.InvalidClassException: com.stickyadstv.web.commands.request.RequestData; incompatible types for field time 6477 java.io.IOException: Broken pipe 1 java.io.IOException: Connection reset by peer 3 java.lang.NullPointerException 821 java.nio.channels.AsynchronousCloseException 12 java.nio.channels.ClosedChannelException 1 - locked <0x00000000a503aa38> (a java.lang.NumberFormatException) 21 org.apache.catalina.connector.CoyoteAdapter$RecycleRequiredException I have uploaded a portion of the catalina log during the test I made: http://www.filedropper.com/catalinatoday11 Thomas 2015-04-21 16:02 GMT+02:00 Christopher Schultz <ch...@christopherschultz.net >: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Thomas, > > On 4/20/15 8:11 AM, Thomas Boniface wrote: > > I have tried to find help regarding an issue we experience with > > our platform leading to random file descriptor peaks. This happens > > more often on heavy load but can also happen on low traffic > > periods. > > > > Our application is using servlet 3.0 async features and an async > > connector. We noticed that a lot of issues regarding asynchronous > > feature were fixed between our production version and the last > > stable build. We decided to give it a try to see if it improves > > things or at least give clues on what can cause the issue; > > Unfortunately it did neither. > > > > The file descriptor peaks and application blocking happens > > frequently with this version when it only happens rarely on > > previous version (tomcat7 7.0.28-4). > > > > Tomcat is behind an nginx server. The tomcat connector used is > > configured as follows: > > > > We use an Nio connector: <Connector port="8080" > > protocol="org.apache.coyote. http11.Http11NioProtocol" > > selectorTimeout="1000" maxThreads="200" maxHttpHeaderSize="16384" > > address="127.0.0.1" redirectPort="8443"/> > > The default maxConnections is 10000, so nginx can open that many > connections before Tomcat starts to refuse them. > > How are you observing the "file descriptor peak"? Are you getting errors > ? > > - -chris > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > Comment: GPGTools - http://gpgtools.org > > iQIcBAEBCAAGBQJVNlhgAAoJEBzwKT+lPKRYMWIQAKWiP7T8HkZq0L9SmqmZr3rv > DrlWBVXBq21M3/e0P3MNEnKp73SbZtq82i6Ib58cjBi6DMQQJTGisdb1WUqNkxWL > f0J0fizl5wwDho9FJzxHzR/uV3Nm67Bx7QzvroEEyAmE/wRXdFFOlq/rSdKWfVDC > jlBE5Seo+AQiURCONEZ9CYHPwm50yeSy9JzGuH1VXcfUTl0NVXS63vOjLp8XeJKO > 68jT6CuY5uzjvv6ZXeES73zvcthkCbF1/Si1KSVshQ/+aXAFDJDuXLLx0D7PWNV7 > N6jxVeHOoTdogYtfVyuOhQ4Xu6d9d9NddKC1ycMBeRfJP/5zG3YXHAbDdwWP8Sc7 > ip9Md6Y+KA089bRhQ92+6kWqWqtxx1Rg1lhRPkY9nOFc5kEFFsWT8NIEIVWtaN80 > zcanU29juMtJK/+Ov/vwyHljTxOikl2So3l19K2bBaCa1pDQ+NeKRQq4KISLlfjB > 05w88zu7uS8gYbf+uiw/TMZte1skT8tR3AD1Ye5XRV22zz+yKy6Z7nAdGg/bvIug > 8ngVWAQ7mxWt6QAtLRMFS4nw+xBNNNRyMYzvFEkZ6d6Wr67SnQXxRKqAIf8nhZ3h > tqAnU0iXlhrdQCcVsMq/iv9lMgHo2x1NBNfeClkIz3XDvgVDJBbHNAr49WmlC5/H > 3xUS2AOTCIJNuK+W1CTm > =i7t2 > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >