-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 David,
On 8/27/20 17:14, David wrote: > Thank you all for the replies! > > On Thu, Aug 27, 2020 at 3:53 PM Christopher Schultz > <ch...@christopherschultz.net> wrote: >> > David, > > On 8/27/20 13:57, David wrote: >>>> On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz >>>> <ch...@christopherschultz.net> wrote: >>>>> >>>> David, >>>> >>>> On 8/27/20 10:48, David wrote: >>>>>>> In the last two weeks I've had two occurrences where a >>>>>>> single CentOS 7 production server hosting a public >>>>>>> webpage has become unresponsive. The first time, all >>>>>>> 300 available "https-jsse-nio-8443" threads were >>>>>>> consumed, with the max age being around 45minutes, and >>>>>>> all in a "S" status. This time all 300 were consumed in >>>>>>> "S" status with the oldest being around ~16minutes. A >>>>>>> restart of Tomcat on both occasions freed these threads >>>>>>> and the website became responsive again. The >>>>>>> connections are post/get methods which shouldn't take >>>>>>> very long at all. >>>>>>> >>>>>>> CPU/MEM/JVM all appear to be within normal operating >>>>>>> limits. I've not had much luck searching for articles >>>>>>> for this behavior nor finding remedies. The default >>>>>>> timeout values are used in both Tomcat and in the >>>>>>> applications that run within as far as I can tell. >>>>>>> Hopefully someone will have some insight on why the >>>>>>> behavior could be occurring, why isn't Tomcat killing >>>>>>> the connections? Even in a RST/ACK status, shouldn't >>>>>>> Tomcat terminate the connection without an ACK from the >>>>>>> client after the default timeout? >>>> >>>> Can you please post: >>>> >>>> 1. Complete Tomcat version >>>>> I can't find anything more granular than 9.0.29, is there >>>>> a command to show a sub patch level? > > 9.0.29 is the patch-level, so that's fine. You are about 10 > versions out of date (~1 year). Any chance for an upgrade? > >> They had to re-dev many apps last year when we upgraded from I >> want to say 1 or 3 or something equally as horrific. Hopefully >> they are forward compatible with the newer releases and if not >> should surely be tackled now before later, I will certainly bring >> this to the table! I've rarely been bitten by an upgrade from foo.bar.x to foo.bar.y. There is a recent caveat if you are using the AJP connector, but you are not so it's not an issue for you. >>>> 2. Connector configuration (possibly redacted) >>>>> This is the 8443 section of the server.xml *8080 is >>>>> available during the outage and I'm able to curl the >>>>> management page to see the 300 used threads, their status, >>>>> and age* <Service name="Catalina"> >>>>> >>>>> [snip] >>>>> >>>>> <Connector port="8080" protocol="HTTP/1.1" >>>>> connectionTimeout="20000" redirectPort="8443" /> [snip] >>>>> <Connector port="8443" >>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" >>>>> maxThreads="300" SSLEnabled="true" > <SSLHostConfig> >>>>> <Certificate >>>>> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >>>>> >>>>> certificateKeystorePassword="redacted" type="RSA" /> >>>>> </SSLHostConfig> </Connector> [snip] <Connector >>>>> port="8443" >>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" >>>>> maxThreads="300" SSLEnabled="true" > <SSLHostConfig >>>>> protocols="TLSv1.2"> <Certificate >>>>> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >>>>> >>>>> certificateKeystorePassword="redacted" type="RSA" /> >>>>> </SSLHostConfig> </Connector> > > What, two connectors on one port? Do you get errors when starting? >> No errors, one is "with HTTP2" should I delete the other former? Well, one of them will succeed in starting the and other one should fail. Did you copy/paste your config without modification? Weird you don't have any errors. Usually you'll get an IOException or whatever binding to the port twice. > I don't see anything obviously problematic in the above > configuration (other than the double-definition of the 8443 > connector). > > 300 tied-up connections (from your initial report) sounds like a > significant number: probably the thread count. >> Yes sir, that's the NIO thread count for the 8443 connector. > > Mark (as is often the case) is right: take some thread dumps next > time everything locks up and see what all those threads are doing. > Often, it's something like everything is awaiting on a db > connection and the db pool has been exhausted or something. > Relatively simple quick-fixes are available for that, and better, > longer-term fixes as well. > >> Mark/Chris is there a way to dump the connector threads >> specifically? Or simply is it all contained as a machine/process >> thread? Sorry I'm not really a Linux guy. Most of the threads in the server will be connector threads. They will have names like https-nio-[port]-exec-[number]. If you get a thread dump[1], you'll get a stack trace from every thread. Rainer wrote a great presentation about them in the context of Tomcat. Feel free to give it a read: http://home.apache.org/~rjung/presentations/2018-06-13-ApacheRoadShow-Ja vaThreadDumps.pdf >>>> Do you have a single F5 or a group of them? >>>>> A group of them, several HA pairs depending on internal or >>>>> external and application. This server is behind one HA >>>>> pair and is a single server. > > Okay. Just remember that each F5 can make some large number of > connections to Tomcat, so you need to make sure you can handle > them. > > This was a much bigger deal back in the BIO days when thread limit > = connection limit, and the thread limit was usually something like > 250 - 300. NIO is much better, and the default connection limit is > 10k which "ought to be enough for anyone"[1]. >> (lol) > >> I'm more used to the 1-1 of the BIO style, which kinda confused >> me when I asked the F5 to truncate >X connections and alert me >> and there were 600+ connections while Tomcat manager stated ~30. >> Then I read what the non-interrupt was about. Yeah, NIO allows Tomcat to accept a large number of connections and have a small number of threads process the work they represent. It's not totally swarm-style processing because (a) the servlet spec makes some guarantees about which thread processes your request and (b) Java doesn't really have the ability to pause execution in one thread and let another thread take it over. If you really want totally asynchronous processing, your application must opt-into it using a special API. So if you have a bog-standard read, process, write style application, then 300 simultaneous requests will be all you can handle. (Unless you raise that limit, of course.) You said something interesting earlier and I want to make sure I understood you correctly. You said that the application locked-up but you were able to use curl to observe something. Can you be really specific about that? Most requests come through port 8443. Which port did you connect to in order to call curl? If it's 8443 than that's suspicious. If it's 8080 then it makes more sense, as there will be a different thread-pool used for each of those connectors. - -chris [1] https://cwiki.apache.org/confluence/display/TOMCAT/HowTo#HowTo-HowdoIobt ainathreaddumpofmyrunningwebapp? -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9IJboACgkQHPApP6U8 pFjMwQ//aTiwmuOChBg1VtNeaFXqieclyTlAYKswe8QNtMqaug93YzPhBOsbXnEp 0bWONHuLFVfFH3ZPZb0JWAvICL/qzUb31d45RBh2JIoytsertkZvpxsqc/OIy6sz TRsqD0qPIdT9jOFRl8zI9kK3j/afSJhWBvSMHG4kxz+g9ZwP159PGaWtyGjd9pP+ YQO76xQVoKpcYDSW/Miiil5L2pMFxIK/gYNZpxxisCDTAGbUZIBri6yGEL+z2S2M bEpT+dJvzUb5qkrmdBlOzRN2cw0vBuBmx4PL+fUu1E5ruRMLelWf5MM5LxHbd7wD SV0C0RXL+hVlWLLXtcqKInFJZtLtcEnBqsu/+n6FlF9s01wL1AFbKauyZe2GpDo3 R+ggSflvnxYTsl1BtTYJexxewb17BkPd2Wj8nJAUIyf7SorQ3rt3btNUnrhNejGe Fi5s1OD72YAdn2KAEAJDHHndVqCVsLK7Yj0ka4EibFvqM6ke1xnWNz2ufba4JoRu qaALnxKAg3yd3eHlt0UiTfbi5LvqJiIspBBV1jOMJnZskzKBb4u1WZYUSoJYUvnO i8KJRhu671KFg8Cqk0B+QQkR1D40yh3ejjwRIUfD9FF4B1HJT10khCh/99Ot1ZPp bnRpq3BU0H+ECN2fp6Yd4CO9mRmx7m24g63l40qMcJfx5KyEwIs= =4r1b -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
