Tested server is Rocky 8. We don't use Ubuntu in prod so I focused on testing Rocky. Mateusz Kempski
On Wed, 24 May 2023 at 12:18, Deepak Goel <deic...@gmail.com> wrote: > > The below test results are on which OS? Ubuntu 18 or 20? > > On Wed, 24 May 2023, 17:17 Mateusz Kempski, <mkemp...@xtm-intl.com.invalid> > wrote: >> >> They are all identical VMs. We can also reproduce this on bigger >> servers. I don't think this is caused by Rocky or Ubuntu config. I can >> see 2 problems during my tests. >> 1. httpd does not add any servers when test is running. It kills >> keepalive connections and logs "all workers busy or dying" but does >> not add any, maybe one at the end of the test. For example if I set >> StartServers to 3 it goes up to 4 at the end of the test even though >> it has ServerLimit set to 120. >> 2. httpd seems to not register that it has free workers even when I >> set StartServers to 120. There are thousands of idle threads and it >> still logs "all workers busy or dying" and kills keepalive >> connections. This can be worked around by setting ThreadsPerChild and >> ThreadLimit much higher and lowering StartServers/ServerLimit >> respectively. >> For example with following settings I can easily process over 1500 >> concurrent connections without errors and keepalive killing: >> ``` >> <IfModule mpm_event_module> >> ThreadsPerChild 200 >> ThreadLimit 200 >> StartServers 10 >> ServerLimit 15 >> MinSpareThreads 75 >> MaxSpareThreads 3000 >> MaxRequestWorkers 3000 >> MaxConnectionsPerChild 0 >> </IfModule> >> ``` >> Why does httpd behaves this way? It seems to not be intended >> considering ThreadsPerChild is limited to 64 by default by >> ThreadLimit. I will try to report a bug to httpd devs as it looks like >> one to me. >> Test results with this config: >> ``` >> ab -k -t 900 -c 1500 -n 1000000 http://rocky/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.11 (be patient) >> Completed 100000 requests >> Completed 200000 requests >> Completed 300000 requests >> Completed 400000 requests >> Completed 500000 requests >> Completed 600000 requests >> Completed 700000 requests >> Completed 800000 requests >> Completed 900000 requests >> Completed 1000000 requests >> Finished 1000000 requests >> >> >> Server Software: Apache/2.4.37 >> Server Hostname: 10.1.3.11 >> Server Port: 80 >> >> Document Path: / >> Document Length: 7620 bytes >> >> Concurrency Level: 1500 >> Time taken for tests: 13.923 seconds >> Complete requests: 1000000 >> Failed requests: 0 >> Write errors: 0 >> Keep-Alive requests: 990813 >> Total transferred: 7919596479 bytes >> HTML transferred: 7620000000 bytes >> Requests per second: 71821.69 [#/sec] (mean) >> Time per request: 20.885 [ms] (mean) >> Time per request: 0.014 [ms] (mean, across all concurrent requests) >> Transfer rate: 555467.60 [Kbytes/sec] received >> >> Connection Times (ms) >> min mean[+/-sd] median max >> Connect: 0 0 14.1 0 1057 >> Processing: 0 20 33.5 16 1483 >> Waiting: 0 17 20.5 15 1343 >> Total: 0 21 36.4 16 1483 >> >> Percentage of the requests served within a certain time (ms) >> 50% 16 >> 66% 20 >> 75% 23 >> 80% 26 >> 90% 27 >> 95% 31 >> 98% 65 >> 99% 221 >> 100% 1483 (longest request) >> ``` >> >> Mateusz Kempski >> >> Mateusz Kempski >> Linux System Administrator >> XTM International Ltd. >> Email: mkemp...@xtm-intl.com >> xtm.cloud >> >> >> >> >> On Wed, 24 May 2023 at 05:57, Deepak Goel <deic...@gmail.com> wrote: >> > >> > >> > >> > On Tue, May 23, 2023 at 6:19 PM Mateusz Kempski >> > <mkemp...@xtm-intl.com.invalid> wrote: >> >> >> >> Ubuntu 20 idle: >> >> ``` >> >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s >> >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s >> >> ``` >> >> ``` >> >> top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01 >> >> Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, >> >> 0.0 st >> >> MiB Mem : 16006.4 total, 15259.6 free, 251.0 used, 495.8 >> >> buff/cache >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem >> >> ``` >> >> Ubuntu 20 during test: >> >> ``` >> >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s >> >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s >> >> ``` >> >> ``` >> >> top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16 >> >> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie >> >> %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si, >> >> 0.1 st >> >> MiB Mem : 16006.4 total, 14672.6 free, 358.1 used, 975.7 >> >> buff/cache >> > >> > >> > 1. The buff/cache in Ubuntu 20 is almost twice as that of Ubuntu 18. From >> > what I have read, this should improve the performance of Ubuntu 20 over >> > Ubuntu 18. However, the reverse is happening. >> > >> > 2. The Disk write in Ubuntu 20 is less than that of Ubuntu 18. This again >> > should improve performance of Ubuntu 20 over Ubuntu 18. Again the reverse >> > is happening. Are you writing anything to the DISK in your test? >> > >> > 3. Is the hardware same for all configurations (cpu, mem, disk iops, >> > network)? >> > >> > 4. I wonder if we can open 'Performance Bugs' in Ubuntu & Rocky forums. >> > >> > >> >> >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15353.9 avail Mem >> >> >> >> ``` >> >> Ubuntu 18 idle: >> >> ``` >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> ``` >> >> ``` >> >> top - 12:46:31 up 1 min, 1 user, load average: 0.56, 0.30, 0.11 >> >> Tasks: 226 total, 1 running, 112 sleeping, 0 stopped, 0 zombie >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, >> >> 0.0 st >> >> KiB Mem : 16422552 total, 15861664 free, 204228 used, 356660 >> >> buff/cache >> >> KiB Swap: 4194300 total, 4194300 free, 0 used. 15936820 avail Mem >> >> ``` >> >> Ubuntu 18 during test: >> >> ``` >> >> top - 12:47:07 up 1 min, 1 user, load average: 0.55, 0.31, 0.12 >> >> Tasks: 241 total, 1 running, 127 sleeping, 0 stopped, 0 zombie >> >> %Cpu(s): 11.3 us, 17.0 sy, 0.0 ni, 68.3 id, 0.0 wa, 0.0 hi, 3.3 si, >> >> 0.1 st >> >> KiB Mem : 16422552 total, 15687480 free, 329456 used, 405616 >> >> buff/cache >> >> KiB Swap: 4194300 total, 4194300 free, 0 used. 15809620 avail Mem >> >> ``` >> >> ``` >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.76 M/s >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> ``` >> >> >> >> >> >> Mateusz Kempski >> >> >> >> On Tue, 23 May 2023 at 12:13, Deepak Goel <deic...@gmail.com> wrote: >> >> > >> >> > can you please post top, iotop results for both apache 18/20? >> >> > >> >> > On Tue, 23 May 2023, 17:33 Mateusz Kempski, >> >> > <mkemp...@xtm-intl.com.invalid> wrote: >> >> >> >> >> >> On Ubuntu 20 with apache in stock settings + my mpm event config: >> >> >> ``` >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://ubuntu20/ >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, >> >> >> http://www.zeustech.net/ >> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> >> >> >> >> Benchmarking 10.1.3.3 (be patient) >> >> >> Completed 100000 requests >> >> >> Completed 200000 requests >> >> >> Completed 300000 requests >> >> >> Completed 400000 requests >> >> >> Completed 500000 requests >> >> >> Completed 600000 requests >> >> >> Completed 700000 requests >> >> >> Completed 800000 requests >> >> >> Completed 900000 requests >> >> >> Completed 1000000 requests >> >> >> Finished 1000000 requests >> >> >> >> >> >> >> >> >> Server Software: Apache/2.4.41 >> >> >> Server Hostname: 10.1.3.3 >> >> >> Server Port: 80 >> >> >> >> >> >> Document Path: / >> >> >> Document Length: 10918 bytes >> >> >> >> >> >> Concurrency Level: 1000 >> >> >> Time taken for tests: 19.528 seconds >> >> >> Complete requests: 1000000 >> >> >> Failed requests: 234371 >> >> >> (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032) >> >> >> Write errors: 0 >> >> >> Keep-Alive requests: 768663 >> >> >> Total transferred: 8630779988 bytes >> >> >> HTML transferred: 8392262634 bytes >> >> >> Requests per second: 51207.31 [#/sec] (mean) >> >> >> Time per request: 19.528 [ms] (mean) >> >> >> Time per request: 0.020 [ms] (mean, across all concurrent >> >> >> requests) >> >> >> Transfer rate: 431600.59 [Kbytes/sec] received >> >> >> >> >> >> Connection Times (ms) >> >> >> min mean[+/-sd] median max >> >> >> Connect: 0 7 69.0 0 3022 >> >> >> Processing: 0 12 22.6 9 2028 >> >> >> Waiting: 0 6 15.8 4 2023 >> >> >> Total: 0 19 73.8 9 3038 >> >> >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> >> 50% 9 >> >> >> 66% 13 >> >> >> 75% 16 >> >> >> 80% 19 >> >> >> 90% 30 >> >> >> 95% 46 >> >> >> 98% 56 >> >> >> 99% 215 >> >> >> 100% 3038 (longest request) >> >> >> ``` >> >> >> >> >> >> On Ubuntu 18 with apache in stock settings + my mpm event config. >> >> >> There is ~15 thousands length fails on the first run after apache >> >> >> restart then it stays in 100-1k range: >> >> >> ``` >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://ubuntu18/ >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, >> >> >> http://www.zeustech.net/ >> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> >> >> >> >> Benchmarking 10.1.3.3 (be patient) >> >> >> Completed 100000 requests >> >> >> Completed 200000 requests >> >> >> Completed 300000 requests >> >> >> Completed 400000 requests >> >> >> Completed 500000 requests >> >> >> Completed 600000 requests >> >> >> Completed 700000 requests >> >> >> Completed 800000 requests >> >> >> Completed 900000 requests >> >> >> Completed 1000000 requests >> >> >> Finished 1000000 requests >> >> >> >> >> >> >> >> >> Server Software: Apache/2.4.29 >> >> >> Server Hostname: 10.1.3.3 >> >> >> Server Port: 80 >> >> >> >> >> >> Document Path: / >> >> >> Document Length: 10918 bytes >> >> >> >> >> >> Concurrency Level: 1000 >> >> >> Time taken for tests: 17.221 seconds >> >> >> Complete requests: 1000000 >> >> >> Failed requests: 636 >> >> >> (Connect: 0, Receive: 0, Length: 636, Exceptions: 0) >> >> >> Write errors: 0 >> >> >> Keep-Alive requests: 990158 >> >> >> Total transferred: 11220454865 bytes >> >> >> HTML transferred: 10911056152 bytes >> >> >> Requests per second: 58067.59 [#/sec] (mean) >> >> >> Time per request: 17.221 [ms] (mean) >> >> >> Time per request: 0.017 [ms] (mean, across all concurrent >> >> >> requests) >> >> >> Transfer rate: 636274.18 [Kbytes/sec] received >> >> >> >> >> >> Connection Times (ms) >> >> >> min mean[+/-sd] median max >> >> >> Connect: 0 0 17.4 0 1020 >> >> >> Processing: 0 17 11.3 17 377 >> >> >> Waiting: 0 8 9.0 9 368 >> >> >> Total: 0 17 21.6 17 1389 >> >> >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> >> 50% 17 >> >> >> 66% 18 >> >> >> 75% 18 >> >> >> 80% 19 >> >> >> 90% 20 >> >> >> 95% 23 >> >> >> 98% 28 >> >> >> 99% 36 >> >> >> 100% 1389 (longest request) >> >> >> ``` >> >> >> >> >> >> Mateusz Kempski >> >> >> On Mon, 22 May 2023 at 13:05, Deepak Goel <deic...@gmail.com> wrote: >> >> >> > >> >> >> > >> >> >> > >> >> >> > On Mon, 22 May 2023, 17:58 Mateusz Kempski, >> >> >> > <mkemp...@xtm-intl.com.invalid> wrote: >> >> >> >> >> >> >> >> @Yann Ylavic: >> >> >> >> There seems to be no difference in configuration file except for >> >> >> >> some comments: >> >> >> >> ``` >> >> >> >> diff -Bbde /etc/httpd/conf/httpd.conf >> >> >> >> /etc/httpd/conf/httpd.conf.rocky >> >> >> >> 8a >> >> >> >> # See the httpd.conf(5) man page for more information on this >> >> >> >> configuration, >> >> >> >> # and httpd.service(8) on using and configuring the httpd service. >> >> >> >> # >> >> >> >> . >> >> >> >> ``` >> >> >> >> I will paste complete Rocky config for completeness at the end of >> >> >> >> this message. >> >> >> >> EnableMMAP and EnableSendfile are set to on, differently to defaults >> >> >> >> specified in comments. This is default system configuration present >> >> >> >> on >> >> >> >> both Rocky 8 and Centos 7. I removed httpd and all config files and >> >> >> >> installed it again to make sure. >> >> >> >> ``` >> >> >> >> # Defaults if commented: EnableMMAP On, EnableSendfile Off >> >> >> >> # >> >> >> >> #EnableMMAP off >> >> >> >> EnableSendfile on >> >> >> >> ``` >> >> >> >> >> >> >> >> Both systems return: >> >> >> >> ``` >> >> >> >> ulimit -n >> >> >> >> 1024 >> >> >> >> ``` >> >> >> >> There is no custom config for httpd: >> >> >> >> ``` >> >> >> >> stat /etc/systemd/system/httpd.service.d >> >> >> >> stat: cannot stat ‘/etc/systemd/system/httpd.service.d’: No such >> >> >> >> file >> >> >> >> or directory >> >> >> >> ``` >> >> >> >> systemd itself seems to not apply any limit on both systems: >> >> >> >> ``` >> >> >> >> grep LimitNOFILE /etc/systemd/system.conf >> >> >> >> #DefaultLimitNOFILE= >> >> >> >> ``` >> >> >> >> >> >> >> >> After enabling trace there is a lot of messenges about too many open >> >> >> >> connections and killing keepalive connections. Example snippet: >> >> >> >> ``` >> >> >> >> [Mon May 22 11:57:40.146451 2023] [mpm_event:debug] [pid 7710:tid >> >> >> >> 139800984155904] event.c(1808): Too many open connections (73), not >> >> >> >> accepting new conns in >> >> >> >> this process >> >> >> >> [Mon May 22 11:57:40.146455 2023] [mpm_event:trace1] [pid 7710:tid >> >> >> >> 139800984155904] event.c(1811): Idle workers: 0 >> >> >> >> [Mon May 22 11:57:40.146457 2023] [mpm_event:trace1] [pid 7710:tid >> >> >> >> 139800984155904] event.c(1574): All workers are busy or dying, will >> >> >> >> close 2 keep-alive co >> >> >> >> nnections >> >> >> >> [Mon May 22 11:57:40.146526 2023] [mpm_event:debug] [pid 7379:tid >> >> >> >> 139801403561728] event.c(492): AH00457: Accepting new connections >> >> >> >> again: 50 active conns ( >> >> >> >> 27 lingering/0 clogged/0 suspended), 7 idle workers >> >> >> >> [Mon May 22 11:57:40.146686 2023] [mpm_event:debug] [pid 7511:tid >> >> >> >> 139800849938176] event.c(1808): Too many open connections (86), not >> >> >> >> accepting new conns in >> >> >> >> this process >> >> >> >> [Mon May 22 11:57:40.146698 2023] [mpm_event:trace1] [pid 7511:tid >> >> >> >> 139800849938176] event.c(1811): Idle workers: 11 >> >> >> >> [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7511:tid >> >> >> >> 139800849938176] event.c(1574): All workers are busy or dying, will >> >> >> >> close 34 keep-alive c >> >> >> >> onnections >> >> >> >> [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7644:tid >> >> >> >> 139801336452864] event.c(1574): All workers are busy or dying, will >> >> >> >> close 2 keep-alive co >> >> >> >> nnections >> >> >> >> [Mon May 22 11:57:40.146812 2023] [mpm_event:debug] [pid 7511:tid >> >> >> >> 139800849938176] event.c(492): AH00457: Accepting new connections >> >> >> >> again: 85 active conns ( >> >> >> >> 52 lingering/0 clogged/0 suspended), 6 idle workers >> >> >> >> [Mon May 22 11:57:40.146955 2023] [mpm_event:debug] [pid 7710:tid >> >> >> >> 139800984155904] event.c(492): AH00457: Accepting new connections >> >> >> >> again: 73 active conns ( >> >> >> >> 58 lingering/0 clogged/0 suspended), 9 idle workers >> >> >> >> [Mon May 22 11:57:40.148250 2023] [mpm_event:debug] [pid 7162:tid >> >> >> >> 139800841545472] event.c(492): AH00457: Accepting new connections >> >> >> >> again: 48 active conns ( >> >> >> >> 25 lingering/0 clogged/0 suspended), 2 idle workers >> >> >> >> [Mon May 22 11:57:40.148562 2023] [mpm_event:debug] [pid 7162:tid >> >> >> >> 139800841545472] event.c(1808): Too many open connections (48), not >> >> >> >> accepting new conns in >> >> >> >> this process >> >> >> >> [Mon May 22 11:57:40.148573 2023] [mpm_event:trace1] [pid 7162:tid >> >> >> >> 139800841545472] event.c(1811): Idle workers: 0 >> >> >> >> [Mon May 22 11:57:40.148921 2023] [mpm_event:debug] [pid 7710:tid >> >> >> >> 139800984155904] event.c(1808): Too many open connections (70), not >> >> >> >> accepting new conns in >> >> >> >> this process >> >> >> >> [Mon May 22 11:57:40.148930 2023] [mpm_event:trace1] [pid 7710:tid >> >> >> >> 139800984155904] event.c(1811): Idle workers: 1 >> >> >> >> [Mon May 22 11:57:40.149594 2023] [mpm_event:debug] [pid 7511:tid >> >> >> >> 139800849938176] event.c(1808): Too many open connections (69), not >> >> >> >> accepting new conns in >> >> >> >> this process >> >> >> >> [Mon May 22 11:57:40.149603 2023] [mpm_event:trace1] [pid 7511:tid >> >> >> >> 139800849938176] event.c(1811): Idle workers: 1 >> >> >> >> [Mon May 22 11:57:40.149630 2023] [mpm_event:debug] [pid 7710:tid >> >> >> >> 139800984155904] event.c(492): AH00457: Accepting new connections >> >> >> >> again: 61 active conns ( >> >> >> >> 38 lingering/0 clogged/0 suspended), 2 idle workers >> >> >> >> [Mon May 22 11:57:40.149776 2023] [mpm_event:debug] [pid 7710:tid >> >> >> >> 139800984155904] event.c(1808): Too many open connections (63), not >> >> >> >> accepting new conns in >> >> >> >> this process >> >> >> >> [Mon May 22 11:57:40.149782 2023] [mpm_event:trace1] [pid 7710:tid >> >> >> >> 139800984155904] event.c(1811): Idle workers: 0 >> >> >> >> [Mon May 22 11:57:40.149882 2023] [mpm_event:debug] [pid 7710:tid >> >> >> >> 139800984155904] event.c(492): AH00457: Accepting new connections >> >> >> >> again: 61 active conns ( >> >> >> >> 38 lingering/0 clogged/0 suspended), 2 idle workers >> >> >> >> [Mon May 22 11:57:40.149913 2023] [mpm_event:debug] [pid 7511:tid >> >> >> >> 139800849938176] event.c(492): AH00457: Accepting new connections >> >> >> >> again: 63 active conns ( >> >> >> >> 40 lingering/0 clogged/0 suspended), 2 idle workers >> >> >> >> [Mon May 22 11:57:40.150210 2023] [mpm_event:debug] [pid 7511:tid >> >> >> >> 139800849938176] event.c(1808): Too many open connections (62), not >> >> >> >> accepting new conns in >> >> >> >> this process >> >> >> >> [Mon May 22 11:57:40.150216 2023] [mpm_event:trace1] [pid 7511:tid >> >> >> >> 139800849938176] event.c(1811): Idle workers: 0 >> >> >> >> ``` >> >> >> >> >> >> >> >> @Deepak Goel: >> >> >> >> 1. Here are result with previously mentioned KeepAlive parameters: >> >> >> >> Rocky 8: >> >> >> >> ``` >> >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/ >> >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, >> >> >> >> http://www.zeustech.net/ >> >> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> >> >> >> >> >> >> Benchmarking 10.1.3.11 (be patient) >> >> >> >> Completed 100000 requests >> >> >> >> Completed 200000 requests >> >> >> >> Completed 300000 requests >> >> >> >> Completed 400000 requests >> >> >> >> Completed 500000 requests >> >> >> >> Completed 600000 requests >> >> >> >> Completed 700000 requests >> >> >> >> Completed 800000 requests >> >> >> >> Completed 900000 requests >> >> >> >> Completed 1000000 requests >> >> >> >> Finished 1000000 requests >> >> >> >> >> >> >> >> >> >> >> >> Server Software: Apache/2.4.37 >> >> >> >> Server Hostname: 10.1.3.11 >> >> >> >> Server Port: 80 >> >> >> >> >> >> >> >> Document Path: / >> >> >> >> Document Length: 7620 bytes >> >> >> >> >> >> >> >> Concurrency Level: 1000 >> >> >> >> Time taken for tests: 17.287 seconds >> >> >> >> Complete requests: 1000000 >> >> >> >> Failed requests: 98218 >> >> >> >> (Connect: 0, Receive: 0, Length: 88397, Exceptions: 9821) >> >> >> >> Write errors: 0 >> >> >> >> Keep-Alive requests: 911603 >> >> >> >> Total transferred: 7214426142 bytes >> >> >> >> HTML transferred: 6946414860 bytes >> >> >> >> Requests per second: 57845.81 [#/sec] (mean) >> >> >> >> Time per request: 17.287 [ms] (mean) >> >> >> >> Time per request: 0.017 [ms] (mean, across all concurrent >> >> >> >> requests) >> >> >> >> Transfer rate: 407543.30 [Kbytes/sec] received >> >> >> >> >> >> >> >> Connection Times (ms) >> >> >> >> min mean[+/-sd] median max >> >> >> >> Connect: 0 4 59.9 0 3015 >> >> >> >> Processing: 0 13 14.5 13 432 >> >> >> >> Waiting: 0 13 14.1 13 432 >> >> >> >> Total: 0 17 63.2 15 3036 >> >> >> >> >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> >> >> 50% 15 >> >> >> >> 66% 17 >> >> >> >> 75% 17 >> >> >> >> 80% 18 >> >> >> >> 90% 21 >> >> >> >> 95% 23 >> >> >> >> 98% 28 >> >> >> >> 99% 36 >> >> >> >> 100% 3036 (longest request) >> >> >> >> ``` >> >> >> >> Centos 7: >> >> >> >> ``` >> >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://10.1.3.3/ >> >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, >> >> >> >> http://www.zeustech.net/ >> >> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> >> >> >> >> >> >> Benchmarking 10.1.3.3 (be patient) >> >> >> >> Completed 100000 requests >> >> >> >> Completed 200000 requests >> >> >> >> Completed 300000 requests >> >> >> >> Completed 400000 requests >> >> >> >> Completed 500000 requests >> >> >> >> Completed 600000 requests >> >> >> >> Completed 700000 requests >> >> >> >> Completed 800000 requests >> >> >> >> Completed 900000 requests >> >> >> >> Completed 1000000 requests >> >> >> >> Finished 1000000 requests >> >> >> >> >> >> >> >> >> >> >> >> Server Software: Apache/2.4.6 >> >> >> >> Server Hostname: 10.1.3.3 >> >> >> >> Server Port: 80 >> >> >> >> >> >> >> >> Document Path: / >> >> >> >> Document Length: 7620 bytes >> >> >> >> >> >> >> >> Concurrency Level: 1000 >> >> >> >> Time taken for tests: 20.731 seconds >> >> >> >> Complete requests: 1000000 >> >> >> >> Failed requests: 14 >> >> >> >> (Connect: 0, Receive: 0, Length: 14, Exceptions: 0) >> >> >> >> Write errors: 0 >> >> >> >> Keep-Alive requests: 999986 >> >> >> >> Total transferred: 7913889204 bytes >> >> >> >> HTML transferred: 7619893320 bytes >> >> >> >> Requests per second: 48236.94 [#/sec] (mean) >> >> >> >> Time per request: 20.731 [ms] (mean) >> >> >> >> Time per request: 0.021 [ms] (mean, across all concurrent >> >> >> >> requests) >> >> >> >> Transfer rate: 372794.72 [Kbytes/sec] received >> >> >> >> >> >> >> >> Connection Times (ms) >> >> >> >> min mean[+/-sd] median max >> >> >> >> Connect: 0 0 13.1 0 1025 >> >> >> >> Processing: 2 21 7.9 20 821 >> >> >> >> Waiting: 0 20 7.8 20 821 >> >> >> >> Total: 2 21 15.7 20 1244 >> >> >> >> >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> >> >> 50% 20 >> >> >> >> 66% 22 >> >> >> >> 75% 23 >> >> >> >> 80% 24 >> >> >> >> 90% 25 >> >> >> >> 95% 27 >> >> >> >> 98% 29 >> >> >> >> 99% 30 >> >> >> >> 100% 1244 (longest request) >> >> >> >> ``` >> >> >> >> 3. iotop results: >> >> >> >> Centos 7 idle: >> >> >> >> ``` >> >> >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s >> >> >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> >> >> ``` >> >> >> >> Rocky 8 idle: >> >> >> >> ``` >> >> >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s >> >> >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> >> >> ``` >> >> >> >> Centos 7 during test: >> >> >> >> ``` >> >> >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.56 M/s >> >> >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> >> >> ``` >> >> >> >> Rocky 8 during test: >> >> >> >> ``` >> >> >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 5.35 M/s >> >> >> > >> >> >> > >> >> >> > The DISK WRITE is higher in Rocky as compared to Centos. >> >> >> > >> >> >> > Is it possible for you to test on Ubuntu 18/20 as well? ( Seems >> >> >> > Rocky algo's are taking a performance hit ) >> >> >> >> >> >> >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> >> >> ``` >> >> >> >> I ran top command without parameters on both systems - looks like >> >> >> >> Centos defaults to KiB and Rocky to MiB. >> >> >> >> >> >> >> >> Complete httpd.conf: >> >> >> >> ``` >> >> >> >> # >> >> >> >> # This is the main Apache HTTP server configuration file. It >> >> >> >> contains the >> >> >> >> # configuration directives that give the server its instructions. >> >> >> >> # See <URL:http://httpd.apache.org/docs/2.4/> for detailed >> >> >> >> information. >> >> >> >> # In particular, see >> >> >> >> # <URL:http://httpd.apache.org/docs/2.4/mod/directives.html> >> >> >> >> # for a discussion of each configuration directive. >> >> >> >> # >> >> >> >> # See the httpd.conf(5) man page for more information on this >> >> >> >> configuration, >> >> >> >> # and httpd.service(8) on using and configuring the httpd service. >> >> >> >> # >> >> >> >> # Do NOT simply read the instructions in here without understanding >> >> >> >> # what they do. They're here only as hints or reminders. If you >> >> >> >> are unsure >> >> >> >> # consult the online docs. You have been warned. >> >> >> >> # >> >> >> >> # Configuration and logfile names: If the filenames you specify for >> >> >> >> many >> >> >> >> # of the server's control files begin with "/" (or "drive:/" for >> >> >> >> Win32), the >> >> >> >> # server will use that explicit path. If the filenames do *not* >> >> >> >> begin >> >> >> >> # with "/", the value of ServerRoot is prepended -- so >> >> >> >> 'log/access_log' >> >> >> >> # with ServerRoot set to '/www' will be interpreted by the >> >> >> >> # server as '/www/log/access_log', where as '/log/access_log' will >> >> >> >> be >> >> >> >> # interpreted as '/log/access_log'. >> >> >> >> >> >> >> >> # >> >> >> >> # ServerRoot: The top of the directory tree under which the server's >> >> >> >> # configuration, error, and log files are kept. >> >> >> >> # >> >> >> >> # Do not add a slash at the end of the directory path. If you point >> >> >> >> # ServerRoot at a non-local disk, be sure to specify a local disk >> >> >> >> on the >> >> >> >> # Mutex directive, if file-based mutexes are used. If you wish to >> >> >> >> share the >> >> >> >> # same ServerRoot for multiple httpd daemons, you will need to >> >> >> >> change at >> >> >> >> # least PidFile. >> >> >> >> # >> >> >> >> ServerRoot "/etc/httpd" >> >> >> >> >> >> >> >> # >> >> >> >> # Listen: Allows you to bind Apache to specific IP addresses and/or >> >> >> >> # ports, instead of the default. See also the <VirtualHost> >> >> >> >> # directive. >> >> >> >> # >> >> >> >> # Change this to Listen on specific IP addresses as shown below to >> >> >> >> # prevent Apache from glomming onto all bound IP addresses. >> >> >> >> # >> >> >> >> #Listen 12.34.56.78:80 >> >> >> >> Listen 80 >> >> >> >> >> >> >> >> # >> >> >> >> # Dynamic Shared Object (DSO) Support >> >> >> >> # >> >> >> >> # To be able to use the functionality of a module which was built >> >> >> >> as a DSO you >> >> >> >> # have to place corresponding `LoadModule' lines at this location >> >> >> >> so the >> >> >> >> # directives contained in it are actually available _before_ they >> >> >> >> are used. >> >> >> >> # Statically compiled modules (those listed by `httpd -l') do not >> >> >> >> need >> >> >> >> # to be loaded here. >> >> >> >> # >> >> >> >> # Example: >> >> >> >> # LoadModule foo_module modules/mod_foo.so >> >> >> >> # >> >> >> >> Include conf.modules.d/*.conf >> >> >> >> >> >> >> >> # >> >> >> >> # If you wish httpd to run as a different user or group, you must >> >> >> >> run >> >> >> >> # httpd as root initially and it will switch. >> >> >> >> # >> >> >> >> # User/Group: The name (or #number) of the user/group to run httpd >> >> >> >> as. >> >> >> >> # It is usually good practice to create a dedicated user and group >> >> >> >> for >> >> >> >> # running httpd, as with most system services. >> >> >> >> # >> >> >> >> User apache >> >> >> >> Group apache >> >> >> >> >> >> >> >> # 'Main' server configuration >> >> >> >> # >> >> >> >> # The directives in this section set up the values used by the >> >> >> >> 'main' >> >> >> >> # server, which responds to any requests that aren't handled by a >> >> >> >> # <VirtualHost> definition. These values also provide defaults for >> >> >> >> # any <VirtualHost> containers you may define later in the file. >> >> >> >> # >> >> >> >> # All of these directives may appear inside <VirtualHost> >> >> >> >> containers, >> >> >> >> # in which case these default settings will be overridden for the >> >> >> >> # virtual host being defined. >> >> >> >> # >> >> >> >> >> >> >> >> # >> >> >> >> # ServerAdmin: Your address, where problems with the server should >> >> >> >> be >> >> >> >> # e-mailed. This address appears on some server-generated pages, >> >> >> >> such >> >> >> >> # as error documents. e.g. ad...@your-domain.com >> >> >> >> # >> >> >> >> ServerAdmin root@localhost >> >> >> >> >> >> >> >> # >> >> >> >> # ServerName gives the name and port that the server uses to >> >> >> >> identify itself. >> >> >> >> # This can often be determined automatically, but we recommend you >> >> >> >> specify >> >> >> >> # it explicitly to prevent problems during startup. >> >> >> >> # >> >> >> >> # If your host doesn't have a registered DNS name, enter its IP >> >> >> >> address here. >> >> >> >> # >> >> >> >> #ServerName www.example.com:80 >> >> >> >> >> >> >> >> # >> >> >> >> # Deny access to the entirety of your server's filesystem. You must >> >> >> >> # explicitly permit access to web content directories in other >> >> >> >> # <Directory> blocks below. >> >> >> >> # >> >> >> >> <Directory /> >> >> >> >> AllowOverride none >> >> >> >> Require all denied >> >> >> >> </Directory> >> >> >> >> >> >> >> >> # >> >> >> >> # Note that from this point forward you must specifically allow >> >> >> >> # particular features to be enabled - so if something's not working >> >> >> >> as >> >> >> >> # you might expect, make sure that you have specifically enabled it >> >> >> >> # below. >> >> >> >> # >> >> >> >> >> >> >> >> # >> >> >> >> # DocumentRoot: The directory out of which you will serve your >> >> >> >> # documents. By default, all requests are taken from this >> >> >> >> directory, but >> >> >> >> # symbolic links and aliases may be used to point to other >> >> >> >> locations. >> >> >> >> # >> >> >> >> DocumentRoot "/var/www/html" >> >> >> >> >> >> >> >> # >> >> >> >> # Relax access to content within /var/www. >> >> >> >> # >> >> >> >> <Directory "/var/www"> >> >> >> >> AllowOverride None >> >> >> >> # Allow open access: >> >> >> >> Require all granted >> >> >> >> </Directory> >> >> >> >> >> >> >> >> # Further relax access to the default document root: >> >> >> >> <Directory "/var/www/html"> >> >> >> >> # >> >> >> >> # Possible values for the Options directive are "None", "All", >> >> >> >> # or any combination of: >> >> >> >> # Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI >> >> >> >> MultiViews >> >> >> >> # >> >> >> >> # Note that "MultiViews" must be named *explicitly* --- "Options >> >> >> >> All" >> >> >> >> # doesn't give it to you. >> >> >> >> # >> >> >> >> # The Options directive is both complicated and important. >> >> >> >> Please see >> >> >> >> # http://httpd.apache.org/docs/2.4/mod/core.html#options >> >> >> >> # for more information. >> >> >> >> # >> >> >> >> Options Indexes FollowSymLinks >> >> >> >> >> >> >> >> # >> >> >> >> # AllowOverride controls what directives may be placed in >> >> >> >> .htaccess files. >> >> >> >> # It can be "All", "None", or any combination of the keywords: >> >> >> >> # Options FileInfo AuthConfig Limit >> >> >> >> # >> >> >> >> AllowOverride None >> >> >> >> >> >> >> >> # >> >> >> >> # Controls who can get stuff from this server. >> >> >> >> # >> >> >> >> Require all granted >> >> >> >> </Directory> >> >> >> >> >> >> >> >> # >> >> >> >> # DirectoryIndex: sets the file that Apache will serve if a >> >> >> >> directory >> >> >> >> # is requested. >> >> >> >> # >> >> >> >> <IfModule dir_module> >> >> >> >> DirectoryIndex index.html >> >> >> >> </IfModule> >> >> >> >> >> >> >> >> # >> >> >> >> # The following lines prevent .htaccess and .htpasswd files from >> >> >> >> being >> >> >> >> # viewed by Web clients. >> >> >> >> # >> >> >> >> <Files ".ht*"> >> >> >> >> Require all denied >> >> >> >> </Files> >> >> >> >> >> >> >> >> # >> >> >> >> # ErrorLog: The location of the error log file. >> >> >> >> # If you do not specify an ErrorLog directive within a <VirtualHost> >> >> >> >> # container, error messages relating to that virtual host will be >> >> >> >> # logged here. If you *do* define an error logfile for a >> >> >> >> <VirtualHost> >> >> >> >> # container, that host's errors will be logged there and not here. >> >> >> >> # >> >> >> >> ErrorLog "logs/error_log" >> >> >> >> >> >> >> >> # >> >> >> >> # LogLevel: Control the number of messages logged to the error_log. >> >> >> >> # Possible values include: debug, info, notice, warn, error, crit, >> >> >> >> # alert, emerg. >> >> >> >> # >> >> >> >> LogLevel warn >> >> >> >> >> >> >> >> <IfModule log_config_module> >> >> >> >> # >> >> >> >> # The following directives define some format nicknames for use >> >> >> >> with >> >> >> >> # a CustomLog directive (see below). >> >> >> >> # >> >> >> >> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" >> >> >> >> \"%{User-Agent}i\"" combined >> >> >> >> LogFormat "%h %l %u %t \"%r\" %>s %b" common >> >> >> >> >> >> >> >> <IfModule logio_module> >> >> >> >> # You need to enable mod_logio.c to use %I and %O >> >> >> >> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" >> >> >> >> \"%{User-Agent}i\" %I %O" combinedio >> >> >> >> </IfModule> >> >> >> >> >> >> >> >> # >> >> >> >> # The location and format of the access logfile (Common Logfile >> >> >> >> Format). >> >> >> >> # If you do not define any access logfiles within a <VirtualHost> >> >> >> >> # container, they will be logged here. Contrariwise, if you *do* >> >> >> >> # define per-<VirtualHost> access logfiles, transactions will be >> >> >> >> # logged therein and *not* in this file. >> >> >> >> # >> >> >> >> #CustomLog "logs/access_log" common >> >> >> >> >> >> >> >> # >> >> >> >> # If you prefer a logfile with access, agent, and referer >> >> >> >> information >> >> >> >> # (Combined Logfile Format) you can use the following directive. >> >> >> >> # >> >> >> >> CustomLog "logs/access_log" combined >> >> >> >> </IfModule> >> >> >> >> >> >> >> >> <IfModule alias_module> >> >> >> >> # >> >> >> >> # Redirect: Allows you to tell clients about documents that used >> >> >> >> to >> >> >> >> # exist in your server's namespace, but do not anymore. The >> >> >> >> client >> >> >> >> # will make a new request for the document at its new location. >> >> >> >> # Example: >> >> >> >> # Redirect permanent /foo http://www.example.com/bar >> >> >> >> >> >> >> >> # >> >> >> >> # Alias: Maps web paths into filesystem paths and is used to >> >> >> >> # access content that does not live under the DocumentRoot. >> >> >> >> # Example: >> >> >> >> # Alias /webpath /full/filesystem/path >> >> >> >> # >> >> >> >> # If you include a trailing / on /webpath then the server will >> >> >> >> # require it to be present in the URL. You will also likely >> >> >> >> # need to provide a <Directory> section to allow access to >> >> >> >> # the filesystem path. >> >> >> >> >> >> >> >> # >> >> >> >> # ScriptAlias: This controls which directories contain server >> >> >> >> scripts. >> >> >> >> # ScriptAliases are essentially the same as Aliases, except that >> >> >> >> # documents in the target directory are treated as applications >> >> >> >> and >> >> >> >> # run by the server when requested rather than as documents sent >> >> >> >> to the >> >> >> >> # client. The same rules about trailing "/" apply to ScriptAlias >> >> >> >> # directives as to Alias. >> >> >> >> # >> >> >> >> ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" >> >> >> >> >> >> >> >> </IfModule> >> >> >> >> >> >> >> >> # >> >> >> >> # "/var/www/cgi-bin" should be changed to whatever your >> >> >> >> ScriptAliased >> >> >> >> # CGI directory exists, if you have that configured. >> >> >> >> # >> >> >> >> <Directory "/var/www/cgi-bin"> >> >> >> >> AllowOverride None >> >> >> >> Options None >> >> >> >> Require all granted >> >> >> >> </Directory> >> >> >> >> >> >> >> >> <IfModule mime_module> >> >> >> >> # >> >> >> >> # TypesConfig points to the file containing the list of mappings >> >> >> >> from >> >> >> >> # filename extension to MIME-type. >> >> >> >> # >> >> >> >> TypesConfig /etc/mime.types >> >> >> >> >> >> >> >> # >> >> >> >> # AddType allows you to add to or override the MIME configuration >> >> >> >> # file specified in TypesConfig for specific file types. >> >> >> >> # >> >> >> >> #AddType application/x-gzip .tgz >> >> >> >> # >> >> >> >> # AddEncoding allows you to have certain browsers uncompress >> >> >> >> # information on the fly. Note: Not all browsers support this. >> >> >> >> # >> >> >> >> #AddEncoding x-compress .Z >> >> >> >> #AddEncoding x-gzip .gz .tgz >> >> >> >> # >> >> >> >> # If the AddEncoding directives above are commented-out, then you >> >> >> >> # probably should define those extensions to indicate media >> >> >> >> types: >> >> >> >> # >> >> >> >> AddType application/x-compress .Z >> >> >> >> AddType application/x-gzip .gz .tgz >> >> >> >> >> >> >> >> # >> >> >> >> # AddHandler allows you to map certain file extensions to >> >> >> >> "handlers": >> >> >> >> # actions unrelated to filetype. These can be either built into >> >> >> >> the server >> >> >> >> # or added with the Action directive (see below) >> >> >> >> # >> >> >> >> # To use CGI scripts outside of ScriptAliased directories: >> >> >> >> # (You will also need to add "ExecCGI" to the "Options" >> >> >> >> directive.) >> >> >> >> # >> >> >> >> #AddHandler cgi-script .cgi >> >> >> >> >> >> >> >> # For type maps (negotiated resources): >> >> >> >> #AddHandler type-map var >> >> >> >> >> >> >> >> # >> >> >> >> # Filters allow you to process content before it is sent to the >> >> >> >> client. >> >> >> >> # >> >> >> >> # To parse .shtml files for server-side includes (SSI): >> >> >> >> # (You will also need to add "Includes" to the "Options" >> >> >> >> directive.) >> >> >> >> # >> >> >> >> AddType text/html .shtml >> >> >> >> AddOutputFilter INCLUDES .shtml >> >> >> >> </IfModule> >> >> >> >> >> >> >> >> # >> >> >> >> # Specify a default charset for all content served; this enables >> >> >> >> # interpretation of all content as UTF-8 by default. To use the >> >> >> >> # default browser choice (ISO-8859-1), or to allow the META tags >> >> >> >> # in HTML content to override this choice, comment out this >> >> >> >> # directive: >> >> >> >> # >> >> >> >> AddDefaultCharset UTF-8 >> >> >> >> >> >> >> >> <IfModule mime_magic_module> >> >> >> >> # >> >> >> >> # The mod_mime_magic module allows the server to use various >> >> >> >> hints from the >> >> >> >> # contents of the file itself to determine its type. The >> >> >> >> MIMEMagicFile >> >> >> >> # directive tells the module where the hint definitions are >> >> >> >> located. >> >> >> >> # >> >> >> >> MIMEMagicFile conf/magic >> >> >> >> </IfModule> >> >> >> >> >> >> >> >> # >> >> >> >> # Customizable error responses come in three flavors: >> >> >> >> # 1) plain text 2) local redirects 3) external redirects >> >> >> >> # >> >> >> >> # Some examples: >> >> >> >> #ErrorDocument 500 "The server made a boo boo." >> >> >> >> #ErrorDocument 404 /missing.html >> >> >> >> #ErrorDocument 404 "/cgi-bin/missing_handler.pl" >> >> >> >> #ErrorDocument 402 http://www.example.com/subscription_info.html >> >> >> >> # >> >> >> >> >> >> >> >> # >> >> >> >> # EnableMMAP and EnableSendfile: On systems that support it, >> >> >> >> # memory-mapping or the sendfile syscall may be used to deliver >> >> >> >> # files. This usually improves server performance, but must >> >> >> >> # be turned off when serving from networked-mounted >> >> >> >> # filesystems or if support for these functions is otherwise >> >> >> >> # broken on your system. >> >> >> >> # Defaults if commented: EnableMMAP On, EnableSendfile Off >> >> >> >> # >> >> >> >> #EnableMMAP off >> >> >> >> EnableSendfile on >> >> >> >> >> >> >> >> # Supplemental configuration >> >> >> >> # >> >> >> >> # Load config files in the "/etc/httpd/conf.d" directory, if any. >> >> >> >> IncludeOptional conf.d/*.conf >> >> >> >> >> >> >> >> <IfModule mpm_event_module> >> >> >> >> ThreadsPerChild 25 >> >> >> >> StartServers 3 >> >> >> >> ServerLimit 120 >> >> >> >> MinSpareThreads 75 >> >> >> >> MaxSpareThreads 3000 >> >> >> >> MaxRequestWorkers 3000 >> >> >> >> MaxConnectionsPerChild 0 >> >> >> >> </IfModule> >> >> >> >> ``` >> >> >> >> >> >> >> >> Mateusz Kempski >> >> >> >> Linux System Administrator >> >> >> >> XTM International Ltd. >> >> >> >> Email: mkemp...@xtm-intl.com >> >> >> >> xtm.cloud >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, 22 May 2023 at 11:16, Deepak Goel <deic...@gmail.com> wrote: >> >> >> >> > >> >> >> >> > Hi >> >> >> >> > >> >> >> >> > 1. Please post the test results completely. (Sorry but, saying >> >> >> >> > there is no difference does not help). >> >> >> >> > 2. The memory used in Rocky is comparatively higher than Centos >> >> >> >> > (About 30Meg or so). Also the buff/cache is approx 10 times >> >> >> >> > higher in Rocky than in Centos. >> >> >> >> > 3. Please also post iotop results too. >> >> >> >> > >> >> >> >> > P.S: I hope the top commands is run identical for both Rocky and >> >> >> >> > Centos. >> >> >> >> > >> >> >> >> > Deepak >> >> >> >> > "The greatness of a nation can be judged by the way its animals >> >> >> >> > are treated - Mahatma Gandhi" >> >> >> >> > >> >> >> >> > +91 73500 12833 >> >> >> >> > deic...@gmail.com >> >> >> >> > >> >> >> >> > Facebook: https://www.facebook.com/deicool >> >> >> >> > LinkedIn: www.linkedin.com/in/deicool >> >> >> >> > >> >> >> >> > "Plant a Tree, Go Green" >> >> >> >> > >> >> >> >> > Make In India : http://www.makeinindia.com/home >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > On Mon, May 22, 2023 at 4:28 PM Mateusz Kempski >> >> >> >> > <mkemp...@xtm-intl.com.invalid> wrote: >> >> >> >> >> >> >> >> >> >> I tested again with settings: >> >> >> >> >> ``` >> >> >> >> >> KeepAliveTimeout 300 >> >> >> >> >> MaxKeepAliveRequests 0 >> >> >> >> >> ``` >> >> >> >> >> but there was no difference in results barring normal diffs >> >> >> >> >> run-to-run. >> >> >> >> >> >> >> >> >> >> Below is the top of top from both servers when idle and during >> >> >> >> >> test. >> >> >> >> >> >> >> >> >> >> Rocky 8 no load: >> >> >> >> >> ``` >> >> >> >> >> top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10 >> >> >> >> >> Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 >> >> >> >> >> zombie >> >> >> >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, >> >> >> >> >> 0.0 si, 0.0 st >> >> >> >> >> MiB Mem : 15824.7 total, 10682.4 free, 283.5 used, 4858.8 >> >> >> >> >> buff/cache >> >> >> >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 >> >> >> >> >> avail Mem >> >> >> >> >> ``` >> >> >> >> >> Rocky 8 during test: >> >> >> >> >> ``` >> >> >> >> >> top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28 >> >> >> >> >> Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 >> >> >> >> >> zombie >> >> >> >> >> %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, >> >> >> >> >> 5.4 si, 0.1 st >> >> >> >> >> MiB Mem : 15824.7 total, 9863.0 free, 529.3 used, 5432.3 >> >> >> >> >> buff/cache >> >> >> >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 >> >> >> >> >> avail Mem >> >> >> >> >> ``` >> >> >> >> >> Centos 7 no load: >> >> >> >> >> ``` >> >> >> >> >> top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00 >> >> >> >> >> Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 >> >> >> >> >> zombie >> >> >> >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, >> >> >> >> >> 0.0 si, 0.0 st >> >> >> >> >> KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 >> >> >> >> >> buff/cache >> >> >> >> >> KiB Swap: 0 total, 0 free, 0 used. 15720740 >> >> >> >> >> avail Mem >> >> >> >> >> ``` >> >> >> >> >> Centos 7 during test: >> >> >> >> >> ``` >> >> >> >> >> top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05 >> >> >> >> >> Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 >> >> >> >> >> zombie >> >> >> >> >> %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, >> >> >> >> >> 3.1 si, 0.1 st >> >> >> >> >> KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 >> >> >> >> >> buff/cache >> >> >> >> >> KiB Swap: 0 total, 0 free, 0 used. 15459544 >> >> >> >> >> avail Mem >> >> >> >> >> ``` >> >> >> >> >> >> >> >> >> >> On Mon, 22 May 2023 at 10:34, Deepak Goel <deic...@gmail.com> >> >> >> >> >> wrote: >> >> >> >> >> > >> >> >> >> >> > Hi >> >> >> >> >> > >> >> >> >> >> > I can see about 8000+ requests have timed out in 'Rocky'. This >> >> >> >> >> > is mostly due to Apache, which is unable to handle the load. >> >> >> >> >> > Is it possible to increase the parameter "KeepAliveTimeout" >> >> >> >> >> > (and other KeepAlive parameters). >> >> >> >> >> > >> >> >> >> >> > Is it also possible for you to post the hardware utilisations >> >> >> >> >> > for the 2 different servers (Centos & Rocky)? >> >> >> >> >> > >> >> >> >> >> > Deepak >> >> >> >> >> > "The greatness of a nation can be judged by the way its >> >> >> >> >> > animals are treated - Mahatma Gandhi" >> >> >> >> >> > >> >> >> >> >> > +91 73500 12833 >> >> >> >> >> > deic...@gmail.com >> >> >> >> >> > >> >> >> >> >> > Facebook: https://www.facebook.com/deicool >> >> >> >> >> > LinkedIn: www.linkedin.com/in/deicool >> >> >> >> >> > >> >> >> >> >> > "Plant a Tree, Go Green" >> >> >> >> >> > >> >> >> >> >> > Make In India : http://www.makeinindia.com/home >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski >> >> >> >> >> > <mkemp...@xtm-intl.com.invalid> wrote: >> >> >> >> >> >> >> >> >> >> >> >> Hi all, >> >> >> >> >> >> I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh >> >> >> >> >> >> Centos 7 >> >> >> >> >> >> install, the other is fresh Rocky 8. I installed httpd (on >> >> >> >> >> >> Centos 7 >> >> >> >> >> >> it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured >> >> >> >> >> >> them to >> >> >> >> >> >> point to the same static default html file and enabled mpm >> >> >> >> >> >> event on >> >> >> >> >> >> Centos (mpm event is default on Rocky). Then I added >> >> >> >> >> >> following options >> >> >> >> >> >> to default config on both servers: >> >> >> >> >> >> ``` >> >> >> >> >> >> <IfModule mpm_event_module> >> >> >> >> >> >> ThreadsPerChild 25 >> >> >> >> >> >> StartServers 3 >> >> >> >> >> >> ServerLimit 120 >> >> >> >> >> >> MinSpareThreads 75 >> >> >> >> >> >> MaxSpareThreads 3000 >> >> >> >> >> >> MaxRequestWorkers 3000 >> >> >> >> >> >> MaxConnectionsPerChild 0 >> >> >> >> >> >> </IfModule> >> >> >> >> >> >> ``` >> >> >> >> >> >> After this is done I performed ab tests with keepalive using >> >> >> >> >> >> different >> >> >> >> >> >> Centos 7 VM in the same local network. Can you help me >> >> >> >> >> >> understand >> >> >> >> >> >> these results? On Centos I am able to complete 1 milion >> >> >> >> >> >> requests at >> >> >> >> >> >> 1000 concurrent connections with little to no errors, however >> >> >> >> >> >> with >> >> >> >> >> >> version 2.4.37 on Rocky I get a lot of failed requests due to >> >> >> >> >> >> length >> >> >> >> >> >> and exceptions. Served content is static so I am assuming >> >> >> >> >> >> this is >> >> >> >> >> >> because keepalive connections are closed by the server. I >> >> >> >> >> >> tried >> >> >> >> >> >> various configurations of KeeAaliveTimeout, >> >> >> >> >> >> MaxKeepAliveRequests, >> >> >> >> >> >> AsyncRequestWorkerFactor and other mpm_event options but I >> >> >> >> >> >> got no >> >> >> >> >> >> better results. The only thing that seems to improve the >> >> >> >> >> >> stability of >> >> >> >> >> >> keepalive connections is setting threads and servers to the >> >> >> >> >> >> moon for >> >> >> >> >> >> example: >> >> >> >> >> >> ``` >> >> >> >> >> >> <IfModule mpm_event_module> >> >> >> >> >> >> ThreadsPerChild 50 >> >> >> >> >> >> StartServers 120 >> >> >> >> >> >> ServerLimit 120 >> >> >> >> >> >> MinSpareThreads 6000 >> >> >> >> >> >> MaxSpareThreads 6000 >> >> >> >> >> >> MaxRequestWorkers 6000 >> >> >> >> >> >> MaxConnectionsPerChild 0 >> >> >> >> >> >> </IfModule> >> >> >> >> >> >> ``` >> >> >> >> >> >> However it stills throws more errors (~8k) than 2.4.6 on the >> >> >> >> >> >> first set >> >> >> >> >> >> of settings. This problem occurs only when using keepalive. >> >> >> >> >> >> There are >> >> >> >> >> >> no errors when using ab without -k option, although speed is >> >> >> >> >> >> lower. I >> >> >> >> >> >> can replicate this issue on newest httpd build from source >> >> >> >> >> >> (2.4.57). >> >> >> >> >> >> What is causing this difference of behavior? How can I achieve >> >> >> >> >> >> performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing >> >> >> >> >> >> much more >> >> >> >> >> >> resources at httpd? These errors seems to be a root cause of >> >> >> >> >> >> a problem >> >> >> >> >> >> we have with 502 errors thrown from downstream reverse proxy >> >> >> >> >> >> server >> >> >> >> >> >> when httpd kills keepalive connection prematurely and the >> >> >> >> >> >> proxy is >> >> >> >> >> >> trying to use this connection. >> >> >> >> >> >> >> >> >> >> >> >> Below results of ab tests: >> >> >> >> >> >> >> >> >> >> >> >> Centos 7 VM: >> >> >> >> >> >> ``` >> >> >> >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://centos/ >> >> >> >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> >> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, >> >> >> >> >> >> http://www.zeustech.net/ >> >> >> >> >> >> Licensed to The Apache Software Foundation, >> >> >> >> >> >> http://www.apache.org/ >> >> >> >> >> >> >> >> >> >> >> >> Benchmarking 10.1.3.3 (be patient) >> >> >> >> >> >> Completed 100000 requests >> >> >> >> >> >> Completed 200000 requests >> >> >> >> >> >> Completed 300000 requests >> >> >> >> >> >> Completed 400000 requests >> >> >> >> >> >> Completed 500000 requests >> >> >> >> >> >> Completed 600000 requests >> >> >> >> >> >> Completed 700000 requests >> >> >> >> >> >> Completed 800000 requests >> >> >> >> >> >> Completed 900000 requests >> >> >> >> >> >> Completed 1000000 requests >> >> >> >> >> >> Finished 1000000 requests >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Server Software: Apache/2.4.6 >> >> >> >> >> >> Server Hostname: 10.1.3.3 >> >> >> >> >> >> Server Port: 80 >> >> >> >> >> >> >> >> >> >> >> >> Document Path: / >> >> >> >> >> >> Document Length: 7620 bytes >> >> >> >> >> >> >> >> >> >> >> >> Concurrency Level: 1000 >> >> >> >> >> >> Time taken for tests: 15.285 seconds >> >> >> >> >> >> Complete requests: 1000000 >> >> >> >> >> >> Failed requests: 67 >> >> >> >> >> >> (Connect: 0, Receive: 0, Length: 67, Exceptions: 0) >> >> >> >> >> >> Write errors: 0 >> >> >> >> >> >> Keep-Alive requests: 990567 >> >> >> >> >> >> Total transferred: 7919057974 bytes >> >> >> >> >> >> HTML transferred: 7619489460 bytes >> >> >> >> >> >> Requests per second: 65422.95 [#/sec] (mean) >> >> >> >> >> >> Time per request: 15.285 [ms] (mean) >> >> >> >> >> >> Time per request: 0.015 [ms] (mean, across all >> >> >> >> >> >> concurrent requests) >> >> >> >> >> >> Transfer rate: 505945.41 [Kbytes/sec] received >> >> >> >> >> >> >> >> >> >> >> >> Connection Times (ms) >> >> >> >> >> >> min mean[+/-sd] median max >> >> >> >> >> >> Connect: 0 0 4.7 0 1042 >> >> >> >> >> >> Processing: 3 15 16.8 13 467 >> >> >> >> >> >> Waiting: 0 14 16.2 13 433 >> >> >> >> >> >> Total: 3 15 17.7 13 1081 >> >> >> >> >> >> >> >> >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> >> >> >> >> 50% 13 >> >> >> >> >> >> 66% 15 >> >> >> >> >> >> 75% 16 >> >> >> >> >> >> 80% 17 >> >> >> >> >> >> 90% 21 >> >> >> >> >> >> 95% 23 >> >> >> >> >> >> 98% 32 >> >> >> >> >> >> 99% 44 >> >> >> >> >> >> 100% 1081 (longest request) >> >> >> >> >> >> ``` >> >> >> >> >> >> >> >> >> >> >> >> Rocky 8 VM: >> >> >> >> >> >> ``` >> >> >> >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/ >> >> >> >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> >> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, >> >> >> >> >> >> http://www.zeustech.net/ >> >> >> >> >> >> Licensed to The Apache Software Foundation, >> >> >> >> >> >> http://www.apache.org/ >> >> >> >> >> >> >> >> >> >> >> >> Benchmarking 10.1.3.11 (be patient) >> >> >> >> >> >> Completed 100000 requests >> >> >> >> >> >> Completed 200000 requests >> >> >> >> >> >> Completed 300000 requests >> >> >> >> >> >> Completed 400000 requests >> >> >> >> >> >> Completed 500000 requests >> >> >> >> >> >> Completed 600000 requests >> >> >> >> >> >> Completed 700000 requests >> >> >> >> >> >> Completed 800000 requests >> >> >> >> >> >> Completed 900000 requests >> >> >> >> >> >> Completed 1000000 requests >> >> >> >> >> >> Finished 1000000 requests >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Server Software: Apache/2.4.37 >> >> >> >> >> >> Server Hostname: 10.1.3.11 >> >> >> >> >> >> Server Port: 80 >> >> >> >> >> >> >> >> >> >> >> >> Document Path: / >> >> >> >> >> >> Document Length: 7620 bytes >> >> >> >> >> >> >> >> >> >> >> >> Concurrency Level: 1000 >> >> >> >> >> >> Time taken for tests: 19.101 seconds >> >> >> >> >> >> Complete requests: 1000000 >> >> >> >> >> >> Failed requests: 93159 >> >> >> >> >> >> (Connect: 0, Receive: 0, Length: 85029, Exceptions: 8130) >> >> >> >> >> >> Write errors: 0 >> >> >> >> >> >> Keep-Alive requests: 912753 >> >> >> >> >> >> Total transferred: 7248228337 bytes >> >> >> >> >> >> HTML transferred: 6973694460 bytes >> >> >> >> >> >> Requests per second: 52352.12 [#/sec] (mean) >> >> >> >> >> >> Time per request: 19.101 [ms] (mean) >> >> >> >> >> >> Time per request: 0.019 [ms] (mean, across all >> >> >> >> >> >> concurrent requests) >> >> >> >> >> >> Transfer rate: 370566.51 [Kbytes/sec] received >> >> >> >> >> >> >> >> >> >> >> >> Connection Times (ms) >> >> >> >> >> >> min mean[+/-sd] median max >> >> >> >> >> >> Connect: 0 4 54.4 0 3022 >> >> >> >> >> >> Processing: 0 15 20.6 13 1400 >> >> >> >> >> >> Waiting: 0 13 17.7 12 1400 >> >> >> >> >> >> Total: 0 19 59.0 13 3048 >> >> >> >> >> >> >> >> >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> >> >> >> >> 50% 13 >> >> >> >> >> >> 66% 15 >> >> >> >> >> >> 75% 17 >> >> >> >> >> >> 80% 19 >> >> >> >> >> >> 90% 24 >> >> >> >> >> >> 95% 41 >> >> >> >> >> >> 98% 53 >> >> >> >> >> >> 99% 106 >> >> >> >> >> >> 100% 3048 (longest request) >> >> >> >> >> >> ``` >> >> >> >> >> >> >> >> >> >> >> >> --- >> >> >> >> >> >> Mateusz Kempski >> >> >> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> >> >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> >> >> >> >> >> For additional commands, e-mail: users-h...@httpd.apache.org >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> >> >> >> >> For additional commands, e-mail: users-h...@httpd.apache.org >> >> >> >> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> >> >> >> For additional commands, e-mail: users-h...@httpd.apache.org >> >> >> >> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> >> >> For additional commands, e-mail: users-h...@httpd.apache.org >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> >> For additional commands, e-mail: users-h...@httpd.apache.org >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> For additional commands, e-mail: users-h...@httpd.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
