-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 To whom it may concern,
On 11/29/17 3:16 PM, TurboChargedDad . wrote: >>> So now all you have to do is upgrade to Tomcat 8.0 or, even >>> better, Tomcat 8.5 :) > That's the plan but it's kind of like pulling teeth. > >>> Can you expand on the "weirdness"? I see you have some more >>> details below but I think you could be more specific. > > Let's say that there are 12 users on a given system all running a > tomcat server that has SSL terminated on the same host. user01 > user02 user03 and son on all the way to user12. Each user has > their own /home/userNN directory. Each user has their own own > environment file in /etc/sysconfig/ '/etc/sysconfig/tomcat7@userNN > . In each of those files contains the various settings that are > required for each user. CATALINA_HOME Java path, PID etc. Each > user starts it's own JVM in a work directory in their home > directory. So... separate Tomcat processes for each user, right? > Now imagine that user10's application starts to experience a > database issue and the app stops responding.. It used to be true > that everyone would stop responding because the AJP connectors were > BIO. That doesn't make any sense: the proxy configuration is much more important than whether or not BIO is being used with Tomcat. If one Tomcat is seized-up (which would happen e.g. if all db connections were used and all BIO threads were blocking), there should be no problem with the other workers on the proxy reaching other Tomcat instances. > Then the HTTP connections would stack up across the board. The > stacking of the HTTP connections was expected given the situation. > Eventually the reverse proxy servers would die from running out of > memory if were didn't get the outage under control quickly enough. Now, /that/ I can understand. But you said you were using prefork. :/ > Now that we switched that we have had 2 outages. In both cases the > only tenants impacted from a performance perspective were the > tenants experiencing the failures. Good: if an application fails, it should expect to suffer downtime, but it shouldn't affect the other applications. Assuming we are talking about separate Tomcat JVM instances. > No other alarms were detected during these outages for any other > tenants. Something odd does happen however. The Apache HTTP > connections rise for everyone along with the offending site. > Please see the shared graph. > > https://photos.app.goo.gl/ZzEgpQUdbv9L84X82 > > This is caclulated by doing a netstat and grepping for EST then > httpd then the AJP port that would have connections passed back to > it. ( sudo -tt > /bin/netstat -ntp | grep EST | grep httpd | grep > ':8125' | wc -l ) So what's :8125? The failing application or one of the good ones? If httpd is over-provisioned (which is usually okay) and Tomcat is stalled, then the number of hung connections from httpd -> Tomcat rising is exactly what I'd expect: the web server can accept more connections so it's doing that. Tomcat can't make progress so everyone waits. > It is the example above that determines the connection counts for > each tenant. > > I cannot for the life of me understand how or why this is > happening.. The only rise in connections should be detected in the > offending application? Right? Yes. What you've shown me is that a failing application experiences high connection counts which makes sense. Unless I'm reading your text above incorrectly. Or the graph? > I can't say beyond a shadow of a doubt that the AJP connector > threads aren't being wonky. I am having trouble getting JMX to > tell me that information through zabbix. How are you querying JMX? - -chris -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlogOnwdHGNocmlzQGNo cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFignBAAjU3tpDu/4YqpMOAy kPKgD9/U8gWBuxJpDbEbMwErPjv0gFGGwYKz/VE0d6of+DEJtVn/ZQKaaTSHJ+f1 FwHmJLYr+H8GI3Nc9dobDOtgE5fyEekqWByHA6r+4jU3hBWjqYzcT+ihcEhHhs1v xy3cOaUd/4G/XON9HtoxsQ3wdiOlBT06GNQpYQNJl3/zlYrS+5I4qoYIazQ21dq9 qpNVRuRID5ltIjbq/G95S+y+srMW2xX7O7XUz520gc546C8p+29hmppEoUTjyxch wVn46UEAkIbWaCOCCAYikUlwLPMPLhBhCE/vldCqjqq1VMtPV5fhhLUxm/kNac26 mvIFZuCrAv2jD+SDpB+trQK696STDYlCNrn3s3mMw+fkWVnIhiI3OZPXDzieD+Bf MXO26uAMwAnjsWuJ6M8u+OGhufWW6YetVpJZpibJWCOcx5NLDkm+7BReeVEn1Vv4 p5zjXT/rePAlhz3T+7oUCCUN/O+X7BPIcjdeoh8mp+W3t30dYWWQJI4hHt+ATuq1 7J1fUzyjJNE7iP5kvxnG5lA3Kc0clobhwyMH8D5dgqwR32Hm+7IPE2PQbtx4SU8O pqNTIdi2l2h1TTAjmey1TtP85GZHsl6Rwy0gP8nRso4UtGZp5EPPsUhrRfPSS85D MD3low32ZefGl1Omb1lDCFSEbDg= =qCJH -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
