I missed some of these messages before.. I apologize. Can I send these to you privately.
On Wed, Oct 4, 2017 at 4:01 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > TCD, > > On 10/4/17 3:45 PM, TurboChargedDad . wrote: > > Perhaps I am not wording my question correctly. > > Can you confirm that the connection-pool exhaustion appears to be > happening on the AJP client (httpd/mod_proxy_ajp) and NOT on the > server (Tomcat/AJP)? > > If so, the problem will likely not improve by switching-over to an > NIO-based connector on the Tomcat side. > > Having said that, the real problem is likely to be simple arithmetic. > Remember this expression: > > Ctc = Nhttpd * Cworkers > > Ctc = Connections Tomcat should be prepared to accept (e.g. Connector > maxConnections) > > Nhttpd = # of httpd servers > Cworkers = total # of connections in httpd connection pool for all > workers(!!) > > Imagine the following scenario: > > Nhttpd = 2 > Cworker = 200 > Ntomcat = 2 > > On httpd server A, we have a connection pool with 200 connections. If > Tomcat A goes down, all 200 connections will go to Tomcat B. If that > happens to both proxies (Tomcat A stops responding), then both proxies > will send all 200 connections to Tomcat B. That means that Tomcat B > needs to be able to support 400 connections, not 200. > > Let's say you now have 5 workers (1 for each application). Each worker > gets its own connection pool, and each connection pool has 200 workers > in it. Now, we have a situation where each httpd instance actually has > 1000 (potential) connections in the connection pool, and if Tomcat A > goes down, Tomcat B must be able to handle 2000 connections (1000 from > httpd A and 1000 from httpd B). > > At some point, you can't provision enough threads to handle all of > those connections. > > The solution (bringing this back around again) is to use NIO, because > you can handle a LOT more connections with a lower number of threads. > NIO doesn't allow you to handle more *concurrent* traffic (in fact, it > makes performance a tiny bit worse than BIO), but it will allow you to > have TONS of idle connections that aren't "wasting" request-processing > threads that are just waiting for another actual request to come > across the wire. > > > As a test I changed the following line in one of the many tomcat > > instances running on the server and bounced it. Old <!-- > > <Connector port="9335" protocol="AJP/1.3" redirectPort="8443" > > maxThreads="300" /> --> New <Connector port="9335" > > protocol="org.apache.coyote.ajp.AjpNioProtocol" redirectPort="8443" > > maxThreads="300" /> > > Yep, that's how to do it. > > > As the docs state I am able to verify that it did in fact switch > > over to NIO. > > > > INFO: Starting ProtocolHandler ["ajp-nio-9335"] > > Good. Now you can handle many idle connections with the same number of > threads. > > > Will running NIO and BIO on the same box have a negative impact? > > No. > > > I am thinking they should all be switched to NIO, this was just a > > test to see if I was understanding what I was reading. > I would recommend NIO in all cases. > > > That being said I suspect there are going to be far more tweaks > > that needs to be applied as there are none to date. > > Hopefully not. A recent Tomcat (which you don't actually have) with a > stock configuration should be fairly well-configured to handle a great > deal of traffic without falling-over. > > > I also know that the HTTPD server is running in prefork mode. > That will pose some other issues for you, mostly the ability to handle > bursts of high concurrency from your clients. You can consider it > out-of-scope for this discussion, though. What we want to do for you > is stop httpd+Tomcat from freaking out and getting stopped-up with > even a small number of users. > > > Which I think leaves me with no control over how many connections > > can be handed back from apache on a site by site basis. > > Probably not on a site-by-site basis, but you can adjust the > connection-pool size on a per-worker basis. For prefork it MUST BE > connection_pool_size=1 (the default for prefork httpd) and for > "worker" and similarly-threaded MPMs the default should be fine to use. > > > Really having hard time explaining to others how BIO could have > > caused the connection pool for another use to become exhausted. > Well... > > If one of your Tomcats locks-up (database is dead; might want to check > to see how the application is accessing that... infinite timeouts can > be a real killer, here), it can tie-up connections from > mod_proxy_ajp's connection pool. But those connections should be > per-worker and shouldn't interfere with each other. Unless you have an > uber-worker that handles everything for all those various Tomcats. > > Can you give us a peek at your worker configuration? You explained it > a bit in your first post, but it might be time for some more details... > > - -chris > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlnVTB0dHGNocmlzQGNo > cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFjbSw/9EyMkmQHZicehVjGg > tueSx+0mgVPEAmrVK/ZFVM2W0zlh4xPjh+O2SS7h3ppfOEB8cMrz3qcmAYEIK8QV > vgBieXP7CPo80/TIklJXFz50fl/T4xEGSwtJ91+U9O10Py/0QzJVRqT8ac+EinQH > fiCmkJKpxwCHxvUyziTGyT/H9xkb885ElXxG/KnoTwJ/r4Fph18I4kEj+KPA/2Gd > lc0i0p1MtSlD+lwsvaPnXgkXPhzUoWTOyUjj3/5/XwqeTfPTuRfz3xPH7TCrjc+S > X6JK2oHGvw4cnkskUtodDzBglnR2py+yV5HN4LXjcTe9zYJlhaqL4/kpmC5Bk4/i > TrQ0lyZmbYsLB6a2hHkNgt2G/mR7w+RKtdcTpArsJZdUWuwrj0zZZG4+el31MJuI > Fr6mjkFTLkFVjEHluQGhv+fa80xhT73kqOIJWPRGUbnLW4QyrJ1dogN5kqg+go8F > uASXqh9XZV/NsKGdmkIh07yllwKpMrPOB7QnXUbNDLWxLg7wiyN77uD1yY9uZgbo > GcbyxH4beGTVYK98SIi6I8J2WgIyGhAR7KN4ZBu+A+a0XqiRYKb41i/O43cW2AN2 > n3QrXFQw2Xeqm9s6Y2OGtK7wSovwJfGbaNYY7f1kOHR03YxfJyoRC5GnLM4BvHJb > RP2Ho5einxJLEpEZ9qRM/6x2qKg= > =36Nx > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
