Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Tested server is Rocky 8. We don't use Ubuntu in prod so I focused on testing Rocky. Mateusz Kempski On Wed, 24 May 2023 at 12:18, Deepak Goel wrote: > > The below test results are on which OS? Ubuntu 18 or 20? > > On Wed, 24 May 2023, 17:17 Mateusz Kempski, > wrote: >> >> They are all identical VMs. We can also reproduce this on bigger >> servers. I don't think this is caused by Rocky or Ubuntu config. I can >> see 2 problems during my tests. >> 1. httpd does not add any servers when test is running. It kills >> keepalive connections and logs "all workers busy or dying" but does >> not add any, maybe one at the end of the test. For example if I set >> StartServers to 3 it goes up to 4 at the end of the test even though >> it has ServerLimit set to 120. >> 2. httpd seems to not register that it has free workers even when I >> set StartServers to 120. There are thousands of idle threads and it >> still logs "all workers busy or dying" and kills keepalive >> connections. This can be worked around by setting ThreadsPerChild and >> ThreadLimit much higher and lowering StartServers/ServerLimit >> respectively. >> For example with following settings I can easily process over 1500 >> concurrent connections without errors and keepalive killing: >> ``` >> >> ThreadsPerChild 200 >> ThreadLimit 200 >> StartServers 10 >> ServerLimit 15 >> MinSpareThreads 75 >> MaxSpareThreads 3000 >> MaxRequestWorkers 3000 >> MaxConnectionsPerChild 0 >> >> ``` >> Why does httpd behaves this way? It seems to not be intended >> considering ThreadsPerChild is limited to 64 by default by >> ThreadLimit. I will try to report a bug to httpd devs as it looks like >> one to me. >> Test results with this config: >> ``` >> ab -k -t 900 -c 1500 -n 100 http://rocky/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.11 (be patient) >> Completed 10 requests >> Completed 20 requests >> Completed 30 requests >> Completed 40 requests >> Completed 50 requests >> Completed 60 requests >> Completed 70 requests >> Completed 80 requests >> Completed 90 requests >> Completed 100 requests >> Finished 100 requests >> >> >> Server Software:Apache/2.4.37 >> Server Hostname:10.1.3.11 >> Server Port:80 >> >> Document Path: / >> Document Length:7620 bytes >> >> Concurrency Level: 1500 >> Time taken for tests: 13.923 seconds >> Complete requests: 100 >> Failed requests:0 >> Write errors: 0 >> Keep-Alive requests:990813 >> Total transferred: 7919596479 bytes >> HTML transferred: 762000 bytes >> Requests per second:71821.69 [#/sec] (mean) >> Time per request: 20.885 [ms] (mean) >> Time per request: 0.014 [ms] (mean, across all concurrent requests) >> Transfer rate: 555467.60 [Kbytes/sec] received >> >> Connection Times (ms) >> min mean[+/-sd] median max >> Connect:00 14.1 01057 >> Processing: 0 20 33.5 161483 >> Waiting:0 17 20.5 151343 >> Total: 0 21 36.4 161483 >> >> Percentage of the requests served within a certain time (ms) >> 50% 16 >> 66% 20 >> 75% 23 >> 80% 26 >> 90% 27 >> 95% 31 >> 98% 65 >> 99%221 >> 100% 1483 (longest request) >> ``` >> >> Mateusz Kempski >> >> Mateusz Kempski >> Linux System Administrator >> XTM International Ltd. >> Email: mkemp...@xtm-intl.com >> xtm.cloud >> >> >> >> >> On Wed, 24 May 2023 at 05:57, Deepak Goel wrote: >> > >> > >> > >> > On Tue, May 23, 2023 at 6:19 PM Mateusz Kempski >> > wrote: >> >> >> >> Ubuntu 20 idle: >> >> ``` >> >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s >> >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s >> >> ``` >> >> ``` >> >> top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01 >> >> Tasks: 239 total, 1 running, 238 sleeping, 0
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Before the test: ``` Apache Server Status for 127.0.0.1 (via 127.0.0.1) Server Version: Apache/2.4.37 (rocky) Server MPM: event Server Built: May 17 2023 16:27:49 -- Current Time: Thursday, 25-May-2023 12:51:03 UTC Restart Time: Thursday, 25-May-2023 12:50:52 UTC Parent Server Config. Generation: 1 Parent Server MPM Generation: 0 Server uptime: 11 seconds Server load: 0.03 0.02 0.00 Total accesses: 0 - Total Traffic: 0 kB - Total Duration: 0 CPU Usage: u.01 s.01 cu0 cs0 - .182% CPU load 0 requests/sec - 0 B/second 1 requests currently being processed, 74 idle workers +-+ | || |Connections| Threads | Async connections| | Slot | PID | Stopping |---+-+| | || | total | accepting | busy | idle | writing | keep-alive | closing | |--++--+---+---+--+--+-++-| | 0| 163535 | no | 0 | yes | 0| 25 | 0 | 0 | 0 | |--++--+---+---+--+--+-++-| | 1| 163536 | no | 0 | yes | 0| 25 | 0 | 0 | 0 | |--++--+---+---+--+--+-++-| | 2| 163537 | no | 0 | yes | 1| 24 | 0 | 0 | 0 | |--++--+---+---+--+--+-++-| | Sum | 3 | 0| 0 | | 1| 74 | 0 | 0 | 0 | +-+ __W_ ___. ..
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
They are all identical VMs. We can also reproduce this on bigger servers. I don't think this is caused by Rocky or Ubuntu config. I can see 2 problems during my tests. 1. httpd does not add any servers when test is running. It kills keepalive connections and logs "all workers busy or dying" but does not add any, maybe one at the end of the test. For example if I set StartServers to 3 it goes up to 4 at the end of the test even though it has ServerLimit set to 120. 2. httpd seems to not register that it has free workers even when I set StartServers to 120. There are thousands of idle threads and it still logs "all workers busy or dying" and kills keepalive connections. This can be worked around by setting ThreadsPerChild and ThreadLimit much higher and lowering StartServers/ServerLimit respectively. For example with following settings I can easily process over 1500 concurrent connections without errors and keepalive killing: ``` ThreadsPerChild 200 ThreadLimit 200 StartServers 10 ServerLimit 15 MinSpareThreads 75 MaxSpareThreads 3000 MaxRequestWorkers 3000 MaxConnectionsPerChild 0 ``` Why does httpd behaves this way? It seems to not be intended considering ThreadsPerChild is limited to 64 by default by ThreadLimit. I will try to report a bug to httpd devs as it looks like one to me. Test results with this config: ``` ab -k -t 900 -c 1500 -n 100 http://rocky/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 10.1.3.11 (be patient) Completed 10 requests Completed 20 requests Completed 30 requests Completed 40 requests Completed 50 requests Completed 60 requests Completed 70 requests Completed 80 requests Completed 90 requests Completed 100 requests Finished 100 requests Server Software:Apache/2.4.37 Server Hostname:10.1.3.11 Server Port:80 Document Path: / Document Length:7620 bytes Concurrency Level: 1500 Time taken for tests: 13.923 seconds Complete requests: 100 Failed requests:0 Write errors: 0 Keep-Alive requests:990813 Total transferred: 7919596479 bytes HTML transferred: 762000 bytes Requests per second:71821.69 [#/sec] (mean) Time per request: 20.885 [ms] (mean) Time per request: 0.014 [ms] (mean, across all concurrent requests) Transfer rate: 555467.60 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 14.1 01057 Processing: 0 20 33.5 161483 Waiting:0 17 20.5 151343 Total: 0 21 36.4 161483 Percentage of the requests served within a certain time (ms) 50% 16 66% 20 75% 23 80% 26 90% 27 95% 31 98% 65 99%221 100% 1483 (longest request) ``` Mateusz Kempski Mateusz Kempski Linux System Administrator XTM International Ltd. Email: mkemp...@xtm-intl.com xtm.cloud On Wed, 24 May 2023 at 05:57, Deepak Goel wrote: > > > > On Tue, May 23, 2023 at 6:19 PM Mateusz Kempski > wrote: >> >> Ubuntu 20 idle: >> ``` >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s >> ``` >> ``` >> top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01 >> Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 >> st >> MiB Mem : 16006.4 total, 15259.6 free,251.0 used,495.8 buff/cache >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem >> ``` >> Ubuntu 20 during test: >> ``` >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s >> ``` >> ``` >> top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16 >> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si, 0.1 >> st >> MiB Mem : 16006.4 total, 14672.6 free,358.1 used,975.7 buff/cache > > > 1. The buff/cache in Ubuntu 20 is almost twice as that of Ubuntu 18. From > what I have read, this should improve the performance of Ubuntu 20 over > Ubuntu 18. However, the reverse is happening. > > 2. The Disk write in Ubuntu 20 is less than that of Ubuntu 18. This again > should improve performance of Ubuntu 20 over Ubuntu 18. Again the reverse is > happening. Are you writing anything to the DISK in your test? > > 3. Is the
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Ubuntu 20 idle: ``` Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s ``` ``` top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01 Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 16006.4 total, 15259.6 free,251.0 used,495.8 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem ``` Ubuntu 20 during test: ``` Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s ``` ``` top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16 Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si, 0.1 st MiB Mem : 16006.4 total, 14672.6 free,358.1 used,975.7 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15353.9 avail Mem ``` Ubuntu 18 idle: ``` Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s ``` ``` top - 12:46:31 up 1 min, 1 user, load average: 0.56, 0.30, 0.11 Tasks: 226 total, 1 running, 112 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16422552 total, 15861664 free, 204228 used, 356660 buff/cache KiB Swap: 4194300 total, 4194300 free,0 used. 15936820 avail Mem ``` Ubuntu 18 during test: ``` top - 12:47:07 up 1 min, 1 user, load average: 0.55, 0.31, 0.12 Tasks: 241 total, 1 running, 127 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.3 us, 17.0 sy, 0.0 ni, 68.3 id, 0.0 wa, 0.0 hi, 3.3 si, 0.1 st KiB Mem : 16422552 total, 15687480 free, 329456 used, 405616 buff/cache KiB Swap: 4194300 total, 4194300 free,0 used. 15809620 avail Mem ``` ``` Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.76 M/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s ``` Mateusz Kempski On Tue, 23 May 2023 at 12:13, Deepak Goel wrote: > > can you please post top, iotop results for both apache 18/20? > > On Tue, 23 May 2023, 17:33 Mateusz Kempski, > wrote: >> >> On Ubuntu 20 with apache in stock settings + my mpm event config: >> ``` >> ab -k -t 900 -c 1000 -n 100 http://ubuntu20/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.3 (be patient) >> Completed 10 requests >> Completed 20 requests >> Completed 30 requests >> Completed 40 requests >> Completed 50 requests >> Completed 60 requests >> Completed 70 requests >> Completed 80 requests >> Completed 90 requests >> Completed 100 requests >> Finished 100 requests >> >> >> Server Software:Apache/2.4.41 >> Server Hostname:10.1.3.3 >> Server Port:80 >> >> Document Path: / >> Document Length:10918 bytes >> >> Concurrency Level: 1000 >> Time taken for tests: 19.528 seconds >> Complete requests: 100 >> Failed requests:234371 >> (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032) >> Write errors: 0 >> Keep-Alive requests:768663 >> Total transferred: 8630779988 bytes >> HTML transferred: 8392262634 bytes >> Requests per second:51207.31 [#/sec] (mean) >> Time per request: 19.528 [ms] (mean) >> Time per request: 0.020 [ms] (mean, across all concurrent requests) >> Transfer rate: 431600.59 [Kbytes/sec] received >> >> Connection Times (ms) >> min mean[+/-sd] median max >> Connect:07 69.0 03022 >> Processing: 0 12 22.6 92028 >> Waiting:06 15.8 42023 >> Total: 0 19 73.8 93038 >> >> Percentage of the requests served within a certain time (ms) >> 50% 9 >> 66% 13 >> 75% 16 >> 80% 19 >> 90% 30 >> 95% 46 >> 98% 56 >> 99%215 >> 100% 3038 (longest request) >> ``` >> >> On Ubuntu 18 with apache in stock settings + my mpm event config. >> There is ~15 thousands length fails on the first run after apache >> restart then it stays in 100-1k range: >> ``` >> ab -k -t 900 -c 1000 -n 100 http://ubuntu
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
On Ubuntu 20 with apache in stock settings + my mpm event config: ``` ab -k -t 900 -c 1000 -n 100 http://ubuntu20/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 10.1.3.3 (be patient) Completed 10 requests Completed 20 requests Completed 30 requests Completed 40 requests Completed 50 requests Completed 60 requests Completed 70 requests Completed 80 requests Completed 90 requests Completed 100 requests Finished 100 requests Server Software:Apache/2.4.41 Server Hostname:10.1.3.3 Server Port:80 Document Path: / Document Length:10918 bytes Concurrency Level: 1000 Time taken for tests: 19.528 seconds Complete requests: 100 Failed requests:234371 (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032) Write errors: 0 Keep-Alive requests:768663 Total transferred: 8630779988 bytes HTML transferred: 8392262634 bytes Requests per second:51207.31 [#/sec] (mean) Time per request: 19.528 [ms] (mean) Time per request: 0.020 [ms] (mean, across all concurrent requests) Transfer rate: 431600.59 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:07 69.0 03022 Processing: 0 12 22.6 92028 Waiting:06 15.8 42023 Total: 0 19 73.8 93038 Percentage of the requests served within a certain time (ms) 50% 9 66% 13 75% 16 80% 19 90% 30 95% 46 98% 56 99%215 100% 3038 (longest request) ``` On Ubuntu 18 with apache in stock settings + my mpm event config. There is ~15 thousands length fails on the first run after apache restart then it stays in 100-1k range: ``` ab -k -t 900 -c 1000 -n 100 http://ubuntu18/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 10.1.3.3 (be patient) Completed 10 requests Completed 20 requests Completed 30 requests Completed 40 requests Completed 50 requests Completed 60 requests Completed 70 requests Completed 80 requests Completed 90 requests Completed 100 requests Finished 100 requests Server Software:Apache/2.4.29 Server Hostname:10.1.3.3 Server Port:80 Document Path: / Document Length:10918 bytes Concurrency Level: 1000 Time taken for tests: 17.221 seconds Complete requests: 100 Failed requests:636 (Connect: 0, Receive: 0, Length: 636, Exceptions: 0) Write errors: 0 Keep-Alive requests:990158 Total transferred: 11220454865 bytes HTML transferred: 10911056152 bytes Requests per second:58067.59 [#/sec] (mean) Time per request: 17.221 [ms] (mean) Time per request: 0.017 [ms] (mean, across all concurrent requests) Transfer rate: 636274.18 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 17.4 01020 Processing: 0 17 11.3 17 377 Waiting:08 9.0 9 368 Total: 0 17 21.6 171389 Percentage of the requests served within a certain time (ms) 50% 17 66% 18 75% 18 80% 19 90% 20 95% 23 98% 28 99% 36 100% 1389 (longest request) ``` Mateusz Kempski On Mon, 22 May 2023 at 13:05, Deepak Goel wrote: > > > > On Mon, 22 May 2023, 17:58 Mateusz Kempski, > wrote: >> >> @Yann Ylavic: >> There seems to be no difference in configuration file except for some >> comments: >> ``` >> diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.rocky >> 8a >> # See the httpd.conf(5) man page for more information on this configuration, >> # and httpd.service(8) on using and configuring the httpd service. >> # >> . >> ``` >> I will paste complete Rocky config for completeness at the end of this >> message. >> EnableMMAP and EnableSendfile are set to on, differently to defaults >> specified in comments. This is default system configuration present on >> both Rocky 8 and Centos 7. I removed httpd and all config files and >> installed it again to make sure. >> ``` >> # Defaults if commented: EnableMMAP On, EnableSendfile Off >> # >> #EnableMMAP off >> EnableSendfile on >> ``` >> >> Both systems return: >> ``` >> ulimit -n >> 1024 >> ``` >> There is no custom config for httpd: >> ``` >> stat /etc/systemd/system/ht
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
located. # MIMEMagicFile conf/magic # # Customizable error responses come in three flavors: # 1) plain text 2) local redirects 3) external redirects # # Some examples: #ErrorDocument 500 "The server made a boo boo." #ErrorDocument 404 /missing.html #ErrorDocument 404 "/cgi-bin/missing_handler.pl" #ErrorDocument 402 http://www.example.com/subscription_info.html # # # EnableMMAP and EnableSendfile: On systems that support it, # memory-mapping or the sendfile syscall may be used to deliver # files. This usually improves server performance, but must # be turned off when serving from networked-mounted # filesystems or if support for these functions is otherwise # broken on your system. # Defaults if commented: EnableMMAP On, EnableSendfile Off # #EnableMMAP off EnableSendfile on # Supplemental configuration # # Load config files in the "/etc/httpd/conf.d" directory, if any. IncludeOptional conf.d/*.conf ThreadsPerChild 25 StartServers 3 ServerLimit 120 MinSpareThreads 75 MaxSpareThreads 3000 MaxRequestWorkers 3000 MaxConnectionsPerChild 0 ``` Mateusz Kempski On Mon, 22 May 2023 at 11:04, Yann Ylavic wrote: > > Hi, > > On Mon, May 22, 2023 at 12:19 PM Mateusz Kempski > wrote: > > > > Then I added following options > > to default config on both servers: > > ``` > > > > ThreadsPerChild 25 > > StartServers 3 > > ServerLimit 120 > > MinSpareThreads 75 > > MaxSpareThreads 3000 > > MaxRequestWorkers 3000 > > MaxConnectionsPerChild 0 > > > > ``` > > What is the difference between the two configurations (besides > identical MPM parameters)? Things like EnableMMAP and EnableSendfile > matter too for instance. > > Do the two systems have the same `ulimit -n` (or LimitNOFILE in > systemd) for httpd? > > Also, do you see errors in the error_log file? Maybe "LogLevel > mpm_event:trace1" could help see what happens while not being too > verbose. > > > Regards; > Yann. > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
low the META tags # in HTML content to override this choice, comment out this # directive: # AddDefaultCharset UTF-8 # # The mod_mime_magic module allows the server to use various hints from the # contents of the file itself to determine its type. The MIMEMagicFile # directive tells the module where the hint definitions are located. # MIMEMagicFile conf/magic # # Customizable error responses come in three flavors: # 1) plain text 2) local redirects 3) external redirects # # Some examples: #ErrorDocument 500 "The server made a boo boo." #ErrorDocument 404 /missing.html #ErrorDocument 404 "/cgi-bin/missing_handler.pl" #ErrorDocument 402 http://www.example.com/subscription_info.html # # # EnableMMAP and EnableSendfile: On systems that support it, # memory-mapping or the sendfile syscall may be used to deliver # files. This usually improves server performance, but must # be turned off when serving from networked-mounted # filesystems or if support for these functions is otherwise # broken on your system. # Defaults if commented: EnableMMAP On, EnableSendfile Off # #EnableMMAP off EnableSendfile on # Supplemental configuration # # Load config files in the "/etc/httpd/conf.d" directory, if any. IncludeOptional conf.d/*.conf ThreadsPerChild 25 StartServers 3 ServerLimit 120 MinSpareThreads 75 MaxSpareThreads 3000 MaxRequestWorkers 3000 MaxConnectionsPerChild 0 ``` Mateusz Kempski Linux System Administrator XTM International Ltd. Email: mkemp...@xtm-intl.com xtm.cloud On Mon, 22 May 2023 at 11:16, Deepak Goel wrote: > > Hi > > 1. Please post the test results completely. (Sorry but, saying there is no > difference does not help). > 2. The memory used in Rocky is comparatively higher than Centos (About 30Meg > or so). Also the buff/cache is approx 10 times higher in Rocky than in Centos. > 3. Please also post iotop results too. > > P.S: I hope the top commands is run identical for both Rocky and Centos. > > Deepak > "The greatness of a nation can be judged by the way its animals are treated - > Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > > On Mon, May 22, 2023 at 4:28 PM Mateusz Kempski > wrote: >> >> I tested again with settings: >> ``` >> KeepAliveTimeout 300 >> MaxKeepAliveRequests 0 >> ``` >> but there was no difference in results barring normal diffs run-to-run. >> >> Below is the top of top from both servers when idle and during test. >> >> Rocky 8 no load: >> ``` >> top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10 >> Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 >> st >> MiB Mem : 15824.7 total, 10682.4 free,283.5 used, 4858.8 buff/cache >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 avail Mem >> ``` >> Rocky 8 during test: >> ``` >> top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28 >> Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 5.4 si, 0.1 >> st >> MiB Mem : 15824.7 total, 9863.0 free,529.3 used, 5432.3 buff/cache >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 avail Mem >> ``` >> Centos 7 no load: >> ``` >> top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00 >> Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 >> st >> KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 buff/cache >> KiB Swap:0 total,0 free,0 used. 15720740 avail Mem >> ``` >> Centos 7 during test: >> ``` >> top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05 >> Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 3.1 si, 0.1 >> st >> KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 buff/cache >> KiB Swap:0 total,0 free, 0 used. 15459544 avail Mem >> ``` >> >> On Mon, 22 May 2023 at 10:34, Deepak Goel wrote: >> > >> > Hi >> > >> > I can see about 8000+ requests have timed out in 'Rocky'. This is mostly >> > due to Apache, which is unable to handle the load. Is it possible to
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
I tested again with settings: ``` KeepAliveTimeout 300 MaxKeepAliveRequests 0 ``` but there was no difference in results barring normal diffs run-to-run. Below is the top of top from both servers when idle and during test. Rocky 8 no load: ``` top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10 Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 15824.7 total, 10682.4 free,283.5 used, 4858.8 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 avail Mem ``` Rocky 8 during test: ``` top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28 Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 5.4 si, 0.1 st MiB Mem : 15824.7 total, 9863.0 free,529.3 used, 5432.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 avail Mem ``` Centos 7 no load: ``` top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 buff/cache KiB Swap:0 total,0 free,0 used. 15720740 avail Mem ``` Centos 7 during test: ``` top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05 Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 zombie %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 3.1 si, 0.1 st KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 buff/cache KiB Swap:0 total,0 free,0 used. 15459544 avail Mem ``` On Mon, 22 May 2023 at 10:34, Deepak Goel wrote: > > Hi > > I can see about 8000+ requests have timed out in 'Rocky'. This is mostly due > to Apache, which is unable to handle the load. Is it possible to increase the > parameter "KeepAliveTimeout" (and other KeepAlive parameters). > > Is it also possible for you to post the hardware utilisations for the 2 > different servers (Centos & Rocky)? > > Deepak > "The greatness of a nation can be judged by the way its animals are treated - > Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski > wrote: >> >> Hi all, >> I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7 >> install, the other is fresh Rocky 8. I installed httpd (on Centos 7 >> it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to >> point to the same static default html file and enabled mpm event on >> Centos (mpm event is default on Rocky). Then I added following options >> to default config on both servers: >> ``` >> >> ThreadsPerChild 25 >> StartServers 3 >> ServerLimit 120 >> MinSpareThreads 75 >> MaxSpareThreads 3000 >> MaxRequestWorkers 3000 >> MaxConnectionsPerChild 0 >> >> ``` >> After this is done I performed ab tests with keepalive using different >> Centos 7 VM in the same local network. Can you help me understand >> these results? On Centos I am able to complete 1 milion requests at >> 1000 concurrent connections with little to no errors, however with >> version 2.4.37 on Rocky I get a lot of failed requests due to length >> and exceptions. Served content is static so I am assuming this is >> because keepalive connections are closed by the server. I tried >> various configurations of KeeAaliveTimeout, MaxKeepAliveRequests, >> AsyncRequestWorkerFactor and other mpm_event options but I got no >> better results. The only thing that seems to improve the stability of >> keepalive connections is setting threads and servers to the moon for >> example: >> ``` >> >> ThreadsPerChild 50 >> StartServers 120 >> ServerLimit 120 >> MinSpareThreads 6000 >> MaxSpareThreads 6000 >> MaxRequestWorkers 6000 >> MaxConnectionsPerChild 0 >> >> ``` >> However it stills throws more errors (~8k) than 2.4.6 on the first set >> of settings. This problem occurs only when using keepalive. There are >> no errors when using ab without -k option, although speed is lower. I >> can replicate this issue on newest httpd build from source (2.4.57). >> What is causing this difference of behavior? How can I achieve >> performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing
[users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
uest: 19.101 [ms] (mean) Time per request: 0.019 [ms] (mean, across all concurrent requests) Transfer rate: 370566.51 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:04 54.4 03022 Processing: 0 15 20.6 131400 Waiting:0 13 17.7 121400 Total: 0 19 59.0 133048 Percentage of the requests served within a certain time (ms) 50% 13 66% 15 75% 17 80% 19 90% 24 95% 41 98% 53 99%106 100% 3048 (longest request) ``` --- Mateusz Kempski - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
