I tested again with settings: ``` KeepAliveTimeout 300 MaxKeepAliveRequests 0 ``` but there was no difference in results barring normal diffs run-to-run.
Below is the top of top from both servers when idle and during test. Rocky 8 no load: ``` top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10 Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 15824.7 total, 10682.4 free, 283.5 used, 4858.8 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 avail Mem ``` Rocky 8 during test: ``` top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28 Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 5.4 si, 0.1 st MiB Mem : 15824.7 total, 9863.0 free, 529.3 used, 5432.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 avail Mem ``` Centos 7 no load: ``` top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 buff/cache KiB Swap: 0 total, 0 free, 0 used. 15720740 avail Mem ``` Centos 7 during test: ``` top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05 Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 zombie %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 3.1 si, 0.1 st KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 buff/cache KiB Swap: 0 total, 0 free, 0 used. 15459544 avail Mem ``` On Mon, 22 May 2023 at 10:34, Deepak Goel <deic...@gmail.com> wrote: > > Hi > > I can see about 8000+ requests have timed out in 'Rocky'. This is mostly due > to Apache, which is unable to handle the load. Is it possible to increase the > parameter "KeepAliveTimeout" (and other KeepAlive parameters). > > Is it also possible for you to post the hardware utilisations for the 2 > different servers (Centos & Rocky)? > > Deepak > "The greatness of a nation can be judged by the way its animals are treated - > Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski > <mkemp...@xtm-intl.com.invalid> wrote: >> >> Hi all, >> I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7 >> install, the other is fresh Rocky 8. I installed httpd (on Centos 7 >> it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to >> point to the same static default html file and enabled mpm event on >> Centos (mpm event is default on Rocky). Then I added following options >> to default config on both servers: >> ``` >> <IfModule mpm_event_module> >> ThreadsPerChild 25 >> StartServers 3 >> ServerLimit 120 >> MinSpareThreads 75 >> MaxSpareThreads 3000 >> MaxRequestWorkers 3000 >> MaxConnectionsPerChild 0 >> </IfModule> >> ``` >> After this is done I performed ab tests with keepalive using different >> Centos 7 VM in the same local network. Can you help me understand >> these results? On Centos I am able to complete 1 milion requests at >> 1000 concurrent connections with little to no errors, however with >> version 2.4.37 on Rocky I get a lot of failed requests due to length >> and exceptions. Served content is static so I am assuming this is >> because keepalive connections are closed by the server. I tried >> various configurations of KeeAaliveTimeout, MaxKeepAliveRequests, >> AsyncRequestWorkerFactor and other mpm_event options but I got no >> better results. The only thing that seems to improve the stability of >> keepalive connections is setting threads and servers to the moon for >> example: >> ``` >> <IfModule mpm_event_module> >> ThreadsPerChild 50 >> StartServers 120 >> ServerLimit 120 >> MinSpareThreads 6000 >> MaxSpareThreads 6000 >> MaxRequestWorkers 6000 >> MaxConnectionsPerChild 0 >> </IfModule> >> ``` >> However it stills throws more errors (~8k) than 2.4.6 on the first set >> of settings. This problem occurs only when using keepalive. There are >> no errors when using ab without -k option, although speed is lower. I >> can replicate this issue on newest httpd build from source (2.4.57). >> What is causing this difference of behavior? How can I achieve >> performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing much more >> resources at httpd? These errors seems to be a root cause of a problem >> we have with 502 errors thrown from downstream reverse proxy server >> when httpd kills keepalive connection prematurely and the proxy is >> trying to use this connection. >> >> Below results of ab tests: >> >> Centos 7 VM: >> ``` >> ab -k -t 900 -c 1000 -n 1000000 http://centos/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.3 (be patient) >> Completed 100000 requests >> Completed 200000 requests >> Completed 300000 requests >> Completed 400000 requests >> Completed 500000 requests >> Completed 600000 requests >> Completed 700000 requests >> Completed 800000 requests >> Completed 900000 requests >> Completed 1000000 requests >> Finished 1000000 requests >> >> >> Server Software: Apache/2.4.6 >> Server Hostname: 10.1.3.3 >> Server Port: 80 >> >> Document Path: / >> Document Length: 7620 bytes >> >> Concurrency Level: 1000 >> Time taken for tests: 15.285 seconds >> Complete requests: 1000000 >> Failed requests: 67 >> (Connect: 0, Receive: 0, Length: 67, Exceptions: 0) >> Write errors: 0 >> Keep-Alive requests: 990567 >> Total transferred: 7919057974 bytes >> HTML transferred: 7619489460 bytes >> Requests per second: 65422.95 [#/sec] (mean) >> Time per request: 15.285 [ms] (mean) >> Time per request: 0.015 [ms] (mean, across all concurrent requests) >> Transfer rate: 505945.41 [Kbytes/sec] received >> >> Connection Times (ms) >> min mean[+/-sd] median max >> Connect: 0 0 4.7 0 1042 >> Processing: 3 15 16.8 13 467 >> Waiting: 0 14 16.2 13 433 >> Total: 3 15 17.7 13 1081 >> >> Percentage of the requests served within a certain time (ms) >> 50% 13 >> 66% 15 >> 75% 16 >> 80% 17 >> 90% 21 >> 95% 23 >> 98% 32 >> 99% 44 >> 100% 1081 (longest request) >> ``` >> >> Rocky 8 VM: >> ``` >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.11 (be patient) >> Completed 100000 requests >> Completed 200000 requests >> Completed 300000 requests >> Completed 400000 requests >> Completed 500000 requests >> Completed 600000 requests >> Completed 700000 requests >> Completed 800000 requests >> Completed 900000 requests >> Completed 1000000 requests >> Finished 1000000 requests >> >> >> Server Software: Apache/2.4.37 >> Server Hostname: 10.1.3.11 >> Server Port: 80 >> >> Document Path: / >> Document Length: 7620 bytes >> >> Concurrency Level: 1000 >> Time taken for tests: 19.101 seconds >> Complete requests: 1000000 >> Failed requests: 93159 >> (Connect: 0, Receive: 0, Length: 85029, Exceptions: 8130) >> Write errors: 0 >> Keep-Alive requests: 912753 >> Total transferred: 7248228337 bytes >> HTML transferred: 6973694460 bytes >> Requests per second: 52352.12 [#/sec] (mean) >> Time per request: 19.101 [ms] (mean) >> Time per request: 0.019 [ms] (mean, across all concurrent requests) >> Transfer rate: 370566.51 [Kbytes/sec] received >> >> Connection Times (ms) >> min mean[+/-sd] median max >> Connect: 0 4 54.4 0 3022 >> Processing: 0 15 20.6 13 1400 >> Waiting: 0 13 17.7 12 1400 >> Total: 0 19 59.0 13 3048 >> >> Percentage of the requests served within a certain time (ms) >> 50% 13 >> 66% 15 >> 75% 17 >> 80% 19 >> 90% 24 >> 95% 41 >> 98% 53 >> 99% 106 >> 100% 3048 (longest request) >> ``` >> >> --- >> Mateusz Kempski >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> For additional commands, e-mail: users-h...@httpd.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
