>> So now all you have to do is upgrade to Tomcat 8.0 or, even better, >>Tomcat 8.5 :) That's the plan but it's kind of like pulling teeth.
>>Can you expand on the "weirdness"? I see you have some more details >>below but I think you could be more specific. Let's say that there are 12 users on a given system all running a tomcat server that has SSL terminated on the same host. user01 user02 user03 and son on all the way to user12. Each user has their own /home/userNN directory. Each user has their own own environment file in /etc/sysconfig/ '/etc/sysconfig/tomcat7@userNN . In each of those files contains the various settings that are required for each user. CATALINA_HOME Java path, PID etc. Each user starts it's own JVM in a work directory in their home directory. Now imagine that user10's application starts to experience a database issue and the app stops responding.. It used to be true that everyone would stop responding because the AJP connectors were BIO. Then the HTTP connections would stack up across the board. The stacking of the HTTP connections was expected given the situation. Eventually the reverse proxy servers would die from running out of memory if were didn't get the outage under control quickly enough. Now that we switched that we have had 2 outages. In both cases the only tenants impacted from a performance perspective were the tenants experiencing the failures. No other alarms were detected during these outages for any other tenants. Something odd does happen however. The Apache HTTP connections rise for everyone along with the offending site. Please see the shared graph. https://photos.app.goo.gl/ZzEgpQUdbv9L84X82 This is caclulated by doing a netstat and grepping for EST then httpd then the AJP port that would have connections passed back to it. ( sudo -tt > /bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l ) tcp 0 0 127.0.0.1:37014 127.0.0.1:8125 ESTABLISHED 5529/httpd tcp 0 0 127.0.0.1:40630 127.0.0.1:8125 ESTABLISHED 29638/httpd tcp 0 0 127.0.0.1:40172 127.0.0.1:8125 ESTABLISHED 28592/httpd tcp 0 0 127.0.0.1:36842 127.0.0.1:8125 ESTABLISHED 5529/httpd tcp 0 0 127.0.0.1:40616 127.0.0.1:8125 ESTABLISHED 29640/httpd tcp 0 0 127.0.0.1:37314 127.0.0.1:8125 ESTABLISHED 20267/httpd tcp 0 0 127.0.0.1:39436 127.0.0.1:8125 ESTABLISHED 29577/httpd tcp 0 0 127.0.0.1:39180 127.0.0.1:8125 ESTABLISHED 25280/httpd tcp 0 0 127.0.0.1:40490 127.0.0.1:8125 ESTABLISHED 29577/httpd tcp 0 0 127.0.0.1:39330 127.0.0.1:8125 ESTABLISHED 29633/httpd tcp 0 0 127.0.0.1:40628 127.0.0.1:8125 ESTABLISHED 29631/httpd tcp 0 0 127.0.0.1:39278 127.0.0.1:8125 ESTABLISHED 28799/httpd tcp 0 0 127.0.0.1:39354 127.0.0.1:8125 ESTABLISHED 29637/httpd tcp 0 0 127.0.0.1:39686 127.0.0.1:8125 ESTABLISHED 29575/httpd tcp 0 0 127.0.0.1:37002 127.0.0.1:8125 ESTABLISHED 8354/httpd tcp 0 0 127.0.0.1:39292 127.0.0.1:8125 ESTABLISHED 29574/httpd tcp 0 0 127.0.0.1:39752 127.0.0.1:8125 ESTABLISHED 29631/httpd tcp 0 0 127.0.0.1:41450 127.0.0.1:8125 ESTABLISHED 29574/httpd tcp 0 0 127.0.0.1:37328 127.0.0.1:8125 ESTABLISHED 20266/httpd tcp 0 0 127.0.0.1:39726 127.0.0.1:8125 ESTABLISHED 28799/httpd It is the example above that determines the connection counts for each tenant. I cannot for the life of me understand how or why this is happening.. The only rise in connections should be detected in the offending application? Right? I can't say beyond a shadow of a doubt that the AJP connector threads aren't being wonky. I am having trouble getting JMX to tell me that information through zabbix. Thoughts? Thanks in advance. On Wed, Nov 29, 2017 at 8:51 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Big Papa, > > On 11/29/17 12:06 AM, TurboChargedDad . wrote: > > So.. Thank you for those help me understand the NIO vs BIO in > > tomcat 7.. > > So now all you have to do is upgrade to Tomcat 8.0 or, even better, > Tomcat 8.5 :) > > > I made those changes things have improved quite a bit. I am still > > experiencing some weirdness that I have tried to understand but > > can't get a handle on it. > > Can you expand on the "weirdness"? I see you have some more details > below but I think you could be more specific. > > > Quick overview.. --Proxies-- Apache Proxies (2) - The end user > > terminates SSL at the proxy/edge The proxies use HTTPS/SSL to > > reverse proxy back to the tomcat server. --/Proxies-- > > > > PXY1 & 2 configs for prefork mode. <IfModule prefork.c> > > StartServers 30 MinSpareServers 15 MaxSpareServers 30 ServerLimit > > 400 MaxClients 400 MaxRequestsPerChild 4000 </IfModule> > > If you want high performance, you have to abandon the prefork model > and move to event. Some modules (e.g. mod_php IIRC) don't work > properly with the event model. Think about using your lb with PHP > running on another server as Jim Riggs suggests[1]. You may get better > performance, stability, and fault-tolerance. > > > --Tomcat server-- (1) Apache terminates SSL over the top of Tomcat > > on the same server. Reverse proxies to the tomcat server using NIO > > AJP connectors. --/Tomcat server-- > > Above you say that you are using HTTPS/SSL to connect httpd -> Tomcat. > If you are using AJP then this is not true. So which is it? Are you > using HTTP or AJP as your protocol? > > > Tomcat apache prefork mode config: <IfModule prefork.c> > > StartServers 8 MinSpareServers 5 MaxSpareServers 20 > > ServerLimit 800 MaxClients 800 MaxRequestsPerChild > > 4000 </IfModule> > > What does "Tomcat apache prefork mode" mean? The above is an httpd > configuration, not a Tomcat one. > > > Typical vhost config for a given tenant would look like this.. > > <someuser.conf> <VirtualHost 10.10.10.26:443 > > <http://10.10.10.26:443/>> ServerAdmin ad...@company.com > > <mailto:ad...@company.com> ServerName somewhere.somedomain.com > > <http://somewhere.somedomain.com/> ProxyPass / > > ajp://localhost:8126/ retry=3 > > Okay, now you are using AJP. I think there's definitely some confusion > here as to what is being configured with what. > > > Typical tomcat connector thread config : <Connector port="8126" > > protocol="org.apache.coyote.ajp.AjpNioProtocol" > > redirectPort="8443" maxThreads="300" /> > > If this is the only <Connector> in Tomcat, then you are 100% using AJP > and not HTTP as your protocol. > > Using NIO is the best practice here IMO. > > > We are operating a multi-tenant environment. As of right now, we > > have somewhere around 20 tomcat instances on a large machine of > > which only a handful are "busy". > > Good. > > > It used to be that when any one of them experienced a blocking > > issue. Every one of them went down. All of their AJP connector > > threads would rise until the system because tomcat was > > unresponsive. > > That would be a capacity-planning problem with the httpd proxies. You > probably didn't do your math correctly. > > > So far that appears for the most part to be addressed... > Good. Maybe your math is better but it may still be wrong. > > > However... When an issue is experienced. The site(s) > > experiencing the issue(s) going down doesn't seem to bring down any > > of the other sites. (w00t! w00t!) > > Good. > > > But the httpd connections for each site all still climb together. > > That shouldn't happen (of course!). > > > (Please see attached graph) Again no outage is experienced buy as > > demonstrated by the graph attached to this message. > > Attachments are stripped. Either post your graph elsewhere or describe > it in words. > > > That graph is from zabbix using a custom metric that checks every > > 3 mins.. It does the following for each virtual host / tomcat > > instances > > > > For user25 : UserParameter=somewebsite.constats,sudo -tt > > /bin/netstat -ntp | grep EST | grep httpd | grep ':8125' | wc -l > > UserParameter=somewebsite2.constats,sudo -tt /bin/netstat -ntp | > > grep EST | grep httpd | grep ':8126' | wc -l > > > > So there is virtually no way they can be getting mixed up. Not to > > mention that there are a few that do not experience a rise in > > connections. > > So the "Weirdness" is that your AJP connection count on the httpd > proxy instances increases across all web servers (or all workers?). > What does mod_proxy's status page say for *each worker*? THAT'S what > you need to compare, not just the total number of connections/threads > on the proxy. > > - -chris > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAloeyX0dHGNocmlzQGNo > cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFgAZQ/+OyyDIEaWzgF5zG1o > amUGjCUackktehlpW9STa5kRhIj9REYT4Cql64Cwqvw8ciZVQXAOsYJBACXFKcfa > fvegRQ03YeLy9LDXhPtsx4Nr+qT17ySiFo/MckEIkxCR9mBbFokUb1bVes9kkYQu > yJjQ7AV8SWDWKGdAkbRk4WTuJ23bvRwZ2g4MNb0sDg5dJEQIOY7JYhlFJQLPm/1a > Yeeo/xRMLfY4FBI0zpA1DAXEwiLyXup4SOztHnoxbK5h0YgrRGMOKvAwZXs5/u/2 > NbiqCnsA80OzUrSXd5sDBYzsuR2yOfnnUMcUJh6LmL8XG1Fh+QeJdCDM2KCC0hJL > HSyqEVfl9EyIXj/Bu0DzU6lL8z5lhxoEYeYBE+pMfJKZ3Skb3EHLImuI/Fwv/+hk > T13o1irPIiqq0N0pTvjLtbJbapNZ9Nz8aIzTBLR3TRY6Cf79jI49QRHBYMsth7e/ > 2Ub+WmkbphcoaC1oJEpXu8fbcSuDrdTAzZ9VSvUFsqpUw5b/9OSB2DjKx5GtnHzR > f8bEg682xw8VKlOV7EsD4RvyILmFqisrrTAfE8/3F3IGMYqXV8Is+8BtdScvUgT5 > MzaLVVyT650pv1NJxBjz/UF8BdhJ2Rpj0r5jhqB64Q1F1ZuNO6aLI2qEwRNgwYMV > IoMgNCcqcB03pYbpuYxHKoX5Cfc= > =imbH > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
