Re: Tomcat Handling close_waits
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Norbert, On 8/16/20 13:16, Norbert Elbanbuena wrote: > I also noticed that while server receives the connection requests, > we are seeing multiple requests from the same sources. Some same > source requests (FIN-WAIT) are all in state while other same > sources requests are in other state (some in FIN-WAIT or close_wait > and some Established). > > Why are we seeing multiple requests from the same source at the > same time? Doesn't each socket request exhaust a thread on the > application? Not always. See the table at the bottom of this page: http://tomcat.apache.org/tomcat-9.0-doc/config/http.html#Connector_Compa rison The NIO and APR connectors are non-blocking while waiting for the "next request", so if a client connects and never explicitly disconnects, you can have a rising connection count without running out of request-processing threads. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl88Xo4ACgkQHPApP6U8 pFjvVBAAneeiMR8ZeEwa2kzymC4PKAAu6ZhiBUjMpGeyfBjgPrOxegQV+9ucyOo/ Ydk++ubxpYu5NRVruvCyhGbKL6x0x8Ds9jUTKvctrMLrmXK2Fxm/+O50Y0Q2ddfr n1QYT9jmlv4s9rybSoBinm6tOCTjZLxExp+f4eAgC6VM5Zfl9Z3QkvUHU+YoV1l6 S4AJteXjZZEytmURxtwYpmtHPVknWvg6vlGAP8FFbH2KAKoaqv0DUB+zWe3vtSoD 0+55j2mJJphQkY00nfqzxEZDgfy68+bcqA+eLPM7ya+DsDjHiaUWOXOFwA8hHAJ3 L7+USQtP28EgEvHU6LsTBwwZxIYo4bCIAiC/Ncs8AdxBUCDiq2dNaS+pESUIkef5 IgFu+mrcZycNxaabmebfiCexIt4Oc+JQzUo1VhGYEuulPic67WyYFeYAgoasSIqc MBAwV6/ahYNfoZwiJWjBPkkGBTNDuVQbk8PNbJ9JmvP0yWVolZKZDr9OLJKVvfSF r576LhEIa0fwdA1x115ipnjYGWzPaobnX7xzVjpNNUXtbISG9gGXut57z7u7Ttdw AEBedOx3vFdmBIn4Rw4y3o8it7hOVlMNQqW6f4LW7+KCDKf3yK/4+dipF7exjh2e +7dBdb0UU2xcLOre6ZRJD3XFp4V3bTYFu3QiZl2LStpJjMsfqmM= =r7Rk -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Handling close_waits
Hi. On 17.08.2020 02:51, Norbert Elbanbuena wrote: Hi Paul, Yes these are some samples, I have about 300 of them getting stuck hourly tcp 761 0 192.168.1.50:58870 74.112.28.109:8011 CLOSE_WAIT tcp0 0 192.168.1.50:56938 192.168.1.50:61616 CLOSE_WAIT tcp0 0 192.168.1.50:56924 192.168.1.50:61616 CLOSE_WAIT tcp0 0 192.168.1.50:56910 192.168.1.50:61616 CLOSE_WAIT tcp0 0 192.168.1.50:56912 192.168.1.50:61616 CLOSE_WAIT tcp6 0 6240 192.168.1.50:44352.11.72.45:47123 CLOSE_WAIT tcp6 32 0 192.168.1.50:44334.209.104.242:13402CLOSE_WAIT tcp6 268 0 192.168.1.50:443108.162.244.28:40864CLOSE_WAIT tcp6 32 0 192.168.1.50:44335.167.185.49:10019 CLOSE_WAIT tcp6 32 0 192.168.1.50:44352.24.48.141:60660 CLOSE_WAIT tcp6 202 0 192.168.1.50:443199.189.191.86:51716CLOSE_WAIT tcp6 202 0 192.168.1.50:443199.189.191.86:1386 CLOSE_WAIT Try forcing a Java Garbage Collection in Tomcat, and check if these CLOSE_WAIT sockets are still there after that. If they are gone, you probably have a server application which leaves these sockets "dangling" and does not properly close them. Other people on this list may be able to provide a simpler way to trigger a GC, but I used a script with "jmxsh", like this : # gc_tomcat.jmxsh # force the target JVM to do a GC, from the jmxsh shell # invoke as : # java -jar jmxsh.jar gc_tomcat.jmxsh # # In the following command, replace the port number (-p) # by the port which has been specified in the parameter : # -Dcom.sun.management.jmxremote.port=x # of the target Java JVM startup line jmx_connect -h 127.0.0.1 -p (port) -U (userid) -P (password) jmx_invoke -m java.lang:type=Memory gc jmx_close for jmxsh, see : - https://github.com/davr/jmxsh - http://isbyr.com/use-jmxsh-jmx-cli-tool-to-troubleshoot-remote-jmx-connection/ Warm regards, Norbert -Original Message- From: Paul Carter-Brown Sent: Sunday, August 16, 2020 2:43 PM To: Tomcat Users List Subject: Re: Tomcat Handling close_waits Hi Norbert, The TCP socket states and timers are managed by the underlying OS and not by Tomcat. Can you paste a netstat -an result so I can see what you mean. Also, is the client using HTTP 1.1 with keep-alive or not? What kind of traffic is this? Paul On Sun, Aug 16, 2020 at 7:16 PM Norbert Elbanbuena wrote: I also noticed that while server receives the connection requests, we are seeing multiple requests from the same sources. Some same source requests (FIN-WAIT) are all in state while other same sources requests are in other state (some in FIN-WAIT or close_wait and some Established). Why are we seeing multiple requests from the same source at the same time? Doesn't each socket request exhaust a thread on the application? Warm regards, Norbert Elbanbuena -Original Message- From: Norbert Elbanbuena Sent: Sunday, August 16, 2020 11:35 AM To: Tomcat Users List Subject: Tomcat Handling close_waits Hi, We are experiencing a flood of close_waits in our server. Interestingly, all of the sessions stuck in close_waits are sourced from Amazon IP addresses. Our second server running the same setup (hardware spec, OS version, Tomcat version, etc.) and had almost the same session count and our application and Tomcat didn't go unresponsive. Is there any tuneable parameter for the APR connector for Tomcat to close the stuck sessions, rather than waiting for the application? Warm regards, Norbert Elbanbuena - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat Handling close_waits
Hi Paul, Yes these are some samples, I have about 300 of them getting stuck hourly tcp 761 0 192.168.1.50:58870 74.112.28.109:8011 CLOSE_WAIT tcp0 0 192.168.1.50:56938 192.168.1.50:61616 CLOSE_WAIT tcp0 0 192.168.1.50:56924 192.168.1.50:61616 CLOSE_WAIT tcp0 0 192.168.1.50:56910 192.168.1.50:61616 CLOSE_WAIT tcp0 0 192.168.1.50:56912 192.168.1.50:61616 CLOSE_WAIT tcp6 0 6240 192.168.1.50:44352.11.72.45:47123 CLOSE_WAIT tcp6 32 0 192.168.1.50:44334.209.104.242:13402CLOSE_WAIT tcp6 268 0 192.168.1.50:443108.162.244.28:40864CLOSE_WAIT tcp6 32 0 192.168.1.50:44335.167.185.49:10019 CLOSE_WAIT tcp6 32 0 192.168.1.50:44352.24.48.141:60660 CLOSE_WAIT tcp6 202 0 192.168.1.50:443199.189.191.86:51716CLOSE_WAIT tcp6 202 0 192.168.1.50:443199.189.191.86:1386 CLOSE_WAIT Warm regards, Norbert -Original Message- From: Paul Carter-Brown Sent: Sunday, August 16, 2020 2:43 PM To: Tomcat Users List Subject: Re: Tomcat Handling close_waits Hi Norbert, The TCP socket states and timers are managed by the underlying OS and not by Tomcat. Can you paste a netstat -an result so I can see what you mean. Also, is the client using HTTP 1.1 with keep-alive or not? What kind of traffic is this? Paul On Sun, Aug 16, 2020 at 7:16 PM Norbert Elbanbuena wrote: > I also noticed that while server receives the connection requests, we > are seeing multiple requests from the same sources. Some same source > requests > (FIN-WAIT) are all in state while other same sources requests are in > other state (some in FIN-WAIT or close_wait and some Established). > > Why are we seeing multiple requests from the same source at the same time? > Doesn't each socket request exhaust a thread on the application? > > > Warm regards, > > Norbert Elbanbuena > > -Original Message- > From: Norbert Elbanbuena > Sent: Sunday, August 16, 2020 11:35 AM > To: Tomcat Users List > Subject: Tomcat Handling close_waits > > Hi, > > We are experiencing a flood of close_waits in our server. > Interestingly, all of the sessions stuck in close_waits are sourced > from Amazon IP addresses. Our second server running the same setup > (hardware spec, OS version, Tomcat version, etc.) and had almost the > same session count and our application and Tomcat didn't go unresponsive. > > Is there any tuneable parameter for the APR connector for Tomcat to > close the stuck sessions, rather than waiting for the application? > > Warm regards, > > Norbert Elbanbuena > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Handling close_waits
Hi Norbert, The TCP socket states and timers are managed by the underlying OS and not by Tomcat. Can you paste a netstat -an result so I can see what you mean. Also, is the client using HTTP 1.1 with keep-alive or not? What kind of traffic is this? Paul On Sun, Aug 16, 2020 at 7:16 PM Norbert Elbanbuena wrote: > I also noticed that while server receives the connection requests, we are > seeing multiple requests from the same sources. Some same source requests > (FIN-WAIT) are all in state while other same sources requests are in other > state (some in FIN-WAIT or close_wait and some Established). > > Why are we seeing multiple requests from the same source at the same time? > Doesn't each socket request exhaust a thread on the application? > > > Warm regards, > > Norbert Elbanbuena > > -Original Message- > From: Norbert Elbanbuena > Sent: Sunday, August 16, 2020 11:35 AM > To: Tomcat Users List > Subject: Tomcat Handling close_waits > > Hi, > > We are experiencing a flood of close_waits in our server. Interestingly, > all of the sessions stuck in close_waits are sourced from Amazon IP > addresses. Our second server running the same setup (hardware spec, OS > version, Tomcat version, etc.) and had almost the same session count and > our application and Tomcat didn't go unresponsive. > > Is there any tuneable parameter for the APR connector for Tomcat to close > the stuck sessions, rather than waiting for the application? > > Warm regards, > > Norbert Elbanbuena > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: Tomcat Handling close_waits
I also noticed that while server receives the connection requests, we are seeing multiple requests from the same sources. Some same source requests (FIN-WAIT) are all in state while other same sources requests are in other state (some in FIN-WAIT or close_wait and some Established). Why are we seeing multiple requests from the same source at the same time? Doesn't each socket request exhaust a thread on the application? Warm regards, Norbert Elbanbuena -Original Message- From: Norbert Elbanbuena Sent: Sunday, August 16, 2020 11:35 AM To: Tomcat Users List Subject: Tomcat Handling close_waits Hi, We are experiencing a flood of close_waits in our server. Interestingly, all of the sessions stuck in close_waits are sourced from Amazon IP addresses. Our second server running the same setup (hardware spec, OS version, Tomcat version, etc.) and had almost the same session count and our application and Tomcat didn't go unresponsive. Is there any tuneable parameter for the APR connector for Tomcat to close the stuck sessions, rather than waiting for the application? Warm regards, Norbert Elbanbuena - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat Handling close_waits
Hi, We are experiencing a flood of close_waits in our server. Interestingly, all of the sessions stuck in close_waits are sourced from Amazon IP addresses. Our second server running the same setup (hardware spec, OS version, Tomcat version, etc.) and had almost the same session count and our application and Tomcat didn't go unresponsive. Is there any tuneable parameter for the APR connector for Tomcat to close the stuck sessions, rather than waiting for the application? Warm regards, Norbert Elbanbuena - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
