We have a setup where we have a Elastic LB that spreads load over two Apache servers A1 and A2. These apache servers render a few php pages and primarily redirects the API requests to tomcat appplication servers T1 and T2 as per the following diagram:
Incoming Request | | | \/ LB /\ / \ / \ A1 A2 |\ /| | \ / | | \ / | | / \ | T1 T2 We have recently started to note delays between apache and tomcat. Here is an example log lines from apache mod_slow log and tomcat access log for the same request: APACHE_MOD_SLOW: VNSdtwoAAJkAACnXb-cAAACJ [06/Feb/2015:16:25:51 +0530] elapsed: 50.58 cpu: 0.00(usr)/0.00(sys) pid: 10711 ip: 10.0.0.153 host: www.example.com:443 reqinfo: GET /data/v1/url?url=test-508324 HTTP/1.1 TOMCAT: [06/Feb/2015:16:26:42 +0530] "GET /data/v1/url?url=test-508324 HTTP/1.1" 200 65 10 Apache says the incoming request came at `06/Feb/2015:16:25:51 +0530` and it took ``50s`` to process the request. Whereas tomcat says it took only ``10ms`` to process the request whereas it received the request at ``06/Feb/2015:16:26:42 +0530``. It means it took nearly 50s for apache to connect and send the whole request to tomcat. Apache is using ``mod_proxy_ajp`` to connect to apache. Here is the configuration: <Proxy balancer://prod> BalancerMember ajp://127.0.0.1:8009 route=jvmRoute-8009 connectiontimeout=1 retry=300 BalancerMember ajp://10.0.0.153:8009 route=jvmRoute-8009 connectiontimeout=1 retry=300 ProxySet lbmethod=byrequests </Proxy> Here is the connector configuration from tomcat: <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="4096" minSpareThreads="25" maxSpareThreads="75"/> As per connectiontimeout value, I am assuming apache shouldn't take more than 1 sec to establish connection. Since both apache and tomcat are both on the same machine, there shouldn't be much time lag once connection is established. If it helps, we are using ``https`` requests. But that, I don'y think has anything to do with this. We have done ``ab`` test in order to compare performance using ``https``, ``http`` and connecting tomcat directly. Here are the stats: ab -n5000 -c5 https://example.com/test/100001 Requests per second: 13.67 [#/sec] (mean) Time per request: 365.851 [ms] (mean) Time per request: 73.170 [ms] (mean, across all concurrent requests) Transfer rate: 79.96 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 236 267 95.5 247 3401 Processing: 83 98 58.6 89 1959 Waiting: 82 96 57.5 87 1959 Total: 319 365 134.0 338 3571 Percentage of the requests served within a certain time (ms) 50% 338 66% 347 75% 356 80% 364 90% 399 95% 477 98% 689 99% 869 100% 3571 (longest request) ab -n5000 -c5 http://example.com/test/100001 Time per request: 186.015 [ms] (mean) Time per request: 37.203 [ms] (mean, across all concurrent requests) Transfer rate: 155.55 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 74 79 33.4 76 1278 Processing: 83 107 82.3 91 3964 Waiting: 82 105 60.9 89 940 Total: 157 186 90.1 168 4042 Percentage of the requests served within a certain time (ms) 50% 168 66% 174 75% 180 80% 184 90% 211 95% 259 98% 379 99% 507 100% 4042 (longest request) ab -n5000 -c5 http://IP:8080/test/100001 Requests per second: 31.32 [#/sec] (mean) Time per request: 159.624 [ms] (mean) Time per request: 31.925 [ms] (mean, across all concurrent requests) Transfer rate: 181.30 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 71 76 68.4 73 3079 Processing: 78 84 13.1 81 594 Waiting: 77 83 6.5 81 185 Total: 149 159 71.2 154 3313 Percentage of the requests served within a certain time (ms) 50% 154 66% 157 75% 160 80% 161 90% 166 95% 171 98% 177 99% 189 100% 3313 (longest request) Following observation leads me to believe this that bad performance is depending on sequence of events because none of the requests in the test performed that badly. Versions: Apache: 2.2.4 Tomcat: 8.0.16 Any ideas where the lag is coming from and how to reduce the same?