It should also be noted that if I bounce one of the larger instances. Everyone suffers during the time to startup. The connection counts raise in the same way although I am not sure at this time if there is an actual outage experienced by anyone. I will have to do some testing to determine that.
On Wed, Nov 29, 2017 at 2:16 PM, TurboChargedDad . <linuxhpc...@gmail.com> wrote: > >> So now all you have to do is upgrade to Tomcat 8.0 or, even better, > >>Tomcat 8.5 :) > That's the plan but it's kind of like pulling teeth. > > >>Can you expand on the "weirdness"? I see you have some more details > >>below but I think you could be more specific. > > Let's say that there are 12 users on a given system all running a tomcat > server that has SSL terminated on the same host. user01 user02 user03 and > son on all the way to user12. Each user has their own /home/userNN > directory. Each user has their own own environment file in /etc/sysconfig/ > '/etc/sysconfig/tomcat7@userNN . In each of those files contains the > various settings that are required for each user. CATALINA_HOME Java path, > PID etc. Each user starts it's own JVM in a work directory in their home > directory. > > Now imagine that user10's application starts to experience a database > issue and the app stops responding.. It used to be true that everyone > would stop responding because the AJP connectors were BIO. Then the HTTP > connections would stack up across the board. The stacking of the HTTP > connections was expected given the situation. Eventually the reverse proxy > servers would die from running out of memory if were didn't get the outage > under control quickly enough. > > Now that we switched that we have had 2 outages. In both cases the only > tenants impacted from a performance perspective were the tenants > experiencing the failures. No other alarms were detected during these > outages for any other tenants. Something odd does happen however. The > Apache HTTP connections rise for everyone along with the offending site. > > Please see the shared graph. > > https://photos.app.goo.gl/ZzEgpQUdbv9L84X82 > > This is caclulated by doing a netstat and grepping for EST then httpd > then the AJP port that would have connections passed back to it. ( sudo > -tt > /bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l ) > > tcp 0 0 127.0.0.1:37014 127.0.0.1:8125 > ESTABLISHED 5529/httpd > tcp 0 0 127.0.0.1:40630 127.0.0.1:8125 > ESTABLISHED 29638/httpd > tcp 0 0 127.0.0.1:40172 127.0.0.1:8125 > ESTABLISHED 28592/httpd > tcp 0 0 127.0.0.1:36842 127.0.0.1:8125 > ESTABLISHED 5529/httpd > tcp 0 0 127.0.0.1:40616 127.0.0.1:8125 > ESTABLISHED 29640/httpd > tcp 0 0 127.0.0.1:37314 127.0.0.1:8125 > ESTABLISHED 20267/httpd > tcp 0 0 127.0.0.1:39436 127.0.0.1:8125 > ESTABLISHED 29577/httpd > tcp 0 0 127.0.0.1:39180 127.0.0.1:8125 > ESTABLISHED 25280/httpd > tcp 0 0 127.0.0.1:40490 127.0.0.1:8125 > ESTABLISHED 29577/httpd > tcp 0 0 127.0.0.1:39330 127.0.0.1:8125 > ESTABLISHED 29633/httpd > tcp 0 0 127.0.0.1:40628 127.0.0.1:8125 > ESTABLISHED 29631/httpd > tcp 0 0 127.0.0.1:39278 127.0.0.1:8125 > ESTABLISHED 28799/httpd > tcp 0 0 127.0.0.1:39354 127.0.0.1:8125 > ESTABLISHED 29637/httpd > tcp 0 0 127.0.0.1:39686 127.0.0.1:8125 > ESTABLISHED 29575/httpd > tcp 0 0 127.0.0.1:37002 127.0.0.1:8125 > ESTABLISHED 8354/httpd > tcp 0 0 127.0.0.1:39292 127.0.0.1:8125 > ESTABLISHED 29574/httpd > tcp 0 0 127.0.0.1:39752 127.0.0.1:8125 > ESTABLISHED 29631/httpd > tcp 0 0 127.0.0.1:41450 127.0.0.1:8125 > ESTABLISHED 29574/httpd > tcp 0 0 127.0.0.1:37328 127.0.0.1:8125 > ESTABLISHED 20266/httpd > tcp 0 0 127.0.0.1:39726 127.0.0.1:8125 > ESTABLISHED 28799/httpd > > > It is the example above that determines the connection counts for each > tenant. > > I cannot for the life of me understand how or why this is happening.. The > only rise in connections should be detected in the offending application? > Right? > > I can't say beyond a shadow of a doubt that the AJP connector threads > aren't being wonky. I am having trouble getting JMX to tell me that > information through zabbix. > > > Thoughts? > > Thanks in advance. > > On Wed, Nov 29, 2017 at 8:51 AM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> Big Papa, >> >> On 11/29/17 12:06 AM, TurboChargedDad . wrote: >> > So.. Thank you for those help me understand the NIO vs BIO in >> > tomcat 7.. >> >> So now all you have to do is upgrade to Tomcat 8.0 or, even better, >> Tomcat 8.5 :) >> >> > I made those changes things have improved quite a bit. I am still >> > experiencing some weirdness that I have tried to understand but >> > can't get a handle on it. >> >> Can you expand on the "weirdness"? I see you have some more details >> below but I think you could be more specific. >> >> > Quick overview.. --Proxies-- Apache Proxies (2) - The end user >> > terminates SSL at the proxy/edge The proxies use HTTPS/SSL to >> > reverse proxy back to the tomcat server. --/Proxies-- >> > >> > PXY1 & 2 configs for prefork mode. <IfModule prefork.c> >> > StartServers 30 MinSpareServers 15 MaxSpareServers 30 ServerLimit >> > 400 MaxClients 400 MaxRequestsPerChild 4000 </IfModule> >> >> If you want high performance, you have to abandon the prefork model >> and move to event. Some modules (e.g. mod_php IIRC) don't work >> properly with the event model. Think about using your lb with PHP >> running on another server as Jim Riggs suggests[1]. You may get better >> performance, stability, and fault-tolerance. >> >> > --Tomcat server-- (1) Apache terminates SSL over the top of Tomcat >> > on the same server. Reverse proxies to the tomcat server using NIO >> > AJP connectors. --/Tomcat server-- >> >> Above you say that you are using HTTPS/SSL to connect httpd -> Tomcat. >> If you are using AJP then this is not true. So which is it? Are you >> using HTTP or AJP as your protocol? >> >> > Tomcat apache prefork mode config: <IfModule prefork.c> >> > StartServers 8 MinSpareServers 5 MaxSpareServers 20 >> > ServerLimit 800 MaxClients 800 MaxRequestsPerChild >> > 4000 </IfModule> >> >> What does "Tomcat apache prefork mode" mean? The above is an httpd >> configuration, not a Tomcat one. >> >> > Typical vhost config for a given tenant would look like this.. >> > <someuser.conf> <VirtualHost 10.10.10.26:443 >> > <http://10.10.10.26:443/>> ServerAdmin ad...@company.com >> > <mailto:ad...@company.com> ServerName somewhere.somedomain.com >> > <http://somewhere.somedomain.com/> ProxyPass / >> > ajp://localhost:8126/ retry=3 >> >> Okay, now you are using AJP. I think there's definitely some confusion >> here as to what is being configured with what. >> >> > Typical tomcat connector thread config : <Connector port="8126" >> > protocol="org.apache.coyote.ajp.AjpNioProtocol" >> > redirectPort="8443" maxThreads="300" /> >> >> If this is the only <Connector> in Tomcat, then you are 100% using AJP >> and not HTTP as your protocol. >> >> Using NIO is the best practice here IMO. >> >> > We are operating a multi-tenant environment. As of right now, we >> > have somewhere around 20 tomcat instances on a large machine of >> > which only a handful are "busy". >> >> Good. >> >> > It used to be that when any one of them experienced a blocking >> > issue. Every one of them went down. All of their AJP connector >> > threads would rise until the system because tomcat was >> > unresponsive. >> >> That would be a capacity-planning problem with the httpd proxies. You >> probably didn't do your math correctly. >> >> > So far that appears for the most part to be addressed... >> Good. Maybe your math is better but it may still be wrong. >> >> > However... When an issue is experienced. The site(s) >> > experiencing the issue(s) going down doesn't seem to bring down any >> > of the other sites. (w00t! w00t!) >> >> Good. >> >> > But the httpd connections for each site all still climb together. >> >> That shouldn't happen (of course!). >> >> > (Please see attached graph) Again no outage is experienced buy as >> > demonstrated by the graph attached to this message. >> >> Attachments are stripped. Either post your graph elsewhere or describe >> it in words. >> >> > That graph is from zabbix using a custom metric that checks every >> > 3 mins.. It does the following for each virtual host / tomcat >> > instances >> > >> > For user25 : UserParameter=somewebsite.constats,sudo -tt >> > /bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l >> > UserParameter=somewebsite2.constats,sudo -tt /bin/netstat -ntp | >> > grep EST | grep httpd | grep ':8126' | wc -l >> > >> > So there is virtually no way they can be getting mixed up. Not to >> > mention that there are a few that do not experience a rise in >> > connections. >> >> So the "Weirdness" is that your AJP connection count on the httpd >> proxy instances increases across all web servers (or all workers?). >> What does mod_proxy's status page say for *each worker*? THAT'S what >> you need to compare, not just the total number of connections/threads >> on the proxy. >> >> - -chris >> -----BEGIN PGP SIGNATURE----- >> Comment: GPGTools - http://gpgtools.org >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> >> iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAloeyX0dHGNocmlzQGNo >> cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFgAZQ/+OyyDIEaWzgF5zG1o >> amUGjCUackktehlpW9STa5kRhIj9REYT4Cql64Cwqvw8ciZVQXAOsYJBACXFKcfa >> fvegRQ03YeLy9LDXhPtsx4Nr+qT17ySiFo/MckEIkxCR9mBbFokUb1bVes9kkYQu >> yJjQ7AV8SWDWKGdAkbRk4WTuJ23bvRwZ2g4MNb0sDg5dJEQIOY7JYhlFJQLPm/1a >> Yeeo/xRMLfY4FBI0zpA1DAXEwiLyXup4SOztHnoxbK5h0YgrRGMOKvAwZXs5/u/2 >> NbiqCnsA80OzUrSXd5sDBYzsuR2yOfnnUMcUJh6LmL8XG1Fh+QeJdCDM2KCC0hJL >> HSyqEVfl9EyIXj/Bu0DzU6lL8z5lhxoEYeYBE+pMfJKZ3Skb3EHLImuI/Fwv/+hk >> T13o1irPIiqq0N0pTvjLtbJbapNZ9Nz8aIzTBLR3TRY6Cf79jI49QRHBYMsth7e/ >> 2Ub+WmkbphcoaC1oJEpXu8fbcSuDrdTAzZ9VSvUFsqpUw5b/9OSB2DjKx5GtnHzR >> f8bEg682xw8VKlOV7EsD4RvyILmFqisrrTAfE8/3F3IGMYqXV8Is+8BtdScvUgT5 >> MzaLVVyT650pv1NJxBjz/UF8BdhJ2Rpj0r5jhqB64Q1F1ZuNO6aLI2qEwRNgwYMV >> IoMgNCcqcB03pYbpuYxHKoX5Cfc= >> =imbH >> -----END PGP SIGNATURE----- >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >
