Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Tested server is Rocky 8. We don't use Ubuntu in prod so I focused on testing Rocky. Mateusz Kempski On Wed, 24 May 2023 at 12:18, Deepak Goel wrote: > > The below test results are on which OS? Ubuntu 18 or 20? > > On Wed, 24 May 2023, 17:17 Mateusz Kempski, > wrote: >> >> They are all identical VMs. We can also reproduce this on bigger >> servers. I don't think this is caused by Rocky or Ubuntu config. I can >> see 2 problems during my tests. >> 1. httpd does not add any servers when test is running. It kills >> keepalive connections and logs "all workers busy or dying" but does >> not add any, maybe one at the end of the test. For example if I set >> StartServers to 3 it goes up to 4 at the end of the test even though >> it has ServerLimit set to 120. >> 2. httpd seems to not register that it has free workers even when I >> set StartServers to 120. There are thousands of idle threads and it >> still logs "all workers busy or dying" and kills keepalive >> connections. This can be worked around by setting ThreadsPerChild and >> ThreadLimit much higher and lowering StartServers/ServerLimit >> respectively. >> For example with following settings I can easily process over 1500 >> concurrent connections without errors and keepalive killing: >> ``` >> >> ThreadsPerChild 200 >> ThreadLimit 200 >> StartServers 10 >> ServerLimit 15 >> MinSpareThreads 75 >> MaxSpareThreads 3000 >> MaxRequestWorkers 3000 >> MaxConnectionsPerChild 0 >> >> ``` >> Why does httpd behaves this way? It seems to not be intended >> considering ThreadsPerChild is limited to 64 by default by >> ThreadLimit. I will try to report a bug to httpd devs as it looks like >> one to me. >> Test results with this config: >> ``` >> ab -k -t 900 -c 1500 -n 100 http://rocky/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.11 (be patient) >> Completed 10 requests >> Completed 20 requests >> Completed 30 requests >> Completed 40 requests >> Completed 50 requests >> Completed 60 requests >> Completed 70 requests >> Completed 80 requests >> Completed 90 requests >> Completed 100 requests >> Finished 100 requests >> >> >> Server Software:Apache/2.4.37 >> Server Hostname:10.1.3.11 >> Server Port:80 >> >> Document Path: / >> Document Length:7620 bytes >> >> Concurrency Level: 1500 >> Time taken for tests: 13.923 seconds >> Complete requests: 100 >> Failed requests:0 >> Write errors: 0 >> Keep-Alive requests:990813 >> Total transferred: 7919596479 bytes >> HTML transferred: 762000 bytes >> Requests per second:71821.69 [#/sec] (mean) >> Time per request: 20.885 [ms] (mean) >> Time per request: 0.014 [ms] (mean, across all concurrent requests) >> Transfer rate: 555467.60 [Kbytes/sec] received >> >> Connection Times (ms) >> min mean[+/-sd] median max >> Connect:00 14.1 01057 >> Processing: 0 20 33.5 161483 >> Waiting:0 17 20.5 151343 >> Total: 0 21 36.4 161483 >> >> Percentage of the requests served within a certain time (ms) >> 50% 16 >> 66% 20 >> 75% 23 >> 80% 26 >> 90% 27 >> 95% 31 >> 98% 65 >> 99%221 >> 100% 1483 (longest request) >> ``` >> >> Mateusz Kempski >> >> Mateusz Kempski >> Linux System Administrator >> XTM International Ltd. >> Email: mkemp...@xtm-intl.com >> xtm.cloud >> >> >> >> >> On Wed, 24 May 2023 at 05:57, Deepak Goel wrote: >> > >> > >> > >> > On Tue, May 23, 2023 at 6:19 PM Mateusz Kempski >> > wrote: >> >> >> >> Ubuntu 20 idle: >> >> ``` >> >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s >> >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s >> >> ``` >> >> ``` >> >> top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01 >> >> Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, >> >> 0.0 st >> >> MiB Mem : 16006.4 total, 15259.6 free,251.0 used,495.8 >> >> buff/cache >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem >> >> ``` >> >> Ubuntu 20 during test: >> >> ``` >> >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s >> >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s >> >> ``` >> >> ``` >> >> top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16 >> >> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie >> >> %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si, >> >> 0.1 st >> >> MiB Mem : 16006.4 total, 14672.6 free,358.1 used,975.7 >> >> buff/cache
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Before the test: ``` Apache Server Status for 127.0.0.1 (via 127.0.0.1) Server Version: Apache/2.4.37 (rocky) Server MPM: event Server Built: May 17 2023 16:27:49 -- Current Time: Thursday, 25-May-2023 12:51:03 UTC Restart Time: Thursday, 25-May-2023 12:50:52 UTC Parent Server Config. Generation: 1 Parent Server MPM Generation: 0 Server uptime: 11 seconds Server load: 0.03 0.02 0.00 Total accesses: 0 - Total Traffic: 0 kB - Total Duration: 0 CPU Usage: u.01 s.01 cu0 cs0 - .182% CPU load 0 requests/sec - 0 B/second 1 requests currently being processed, 74 idle workers +-+ | || |Connections| Threads | Async connections| | Slot | PID | Stopping |---+-+| | || | total | accepting | busy | idle | writing | keep-alive | closing | |--++--+---+---+--+--+-++-| | 0| 163535 | no | 0 | yes | 0| 25 | 0 | 0 | 0 | |--++--+---+---+--+--+-++-| | 1| 163536 | no | 0 | yes | 0| 25 | 0 | 0 | 0 | |--++--+---+---+--+--+-++-| | 2| 163537 | no | 0 | yes | 1| 24 | 0 | 0 | 0 | |--++--+---+---+--+--+-++-| | Sum | 3 | 0| 0 | | 1| 74 | 0 | 0 | 0 | +-+ __W_ ___.
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
On Wed, May 24, 2023 at 7:46 AM Mateusz Kempski wrote: > > They are all identical VMs. We can also reproduce this on bigger > servers. I don't think this is caused by Rocky or Ubuntu config. I can > see 2 problems during my tests. > 1. httpd does not add any servers when test is running. It kills > keepalive connections and logs "all workers busy or dying" but does > not add any, maybe one at the end of the test. For example if I set > StartServers to 3 it goes up to 4 at the end of the test even though > it has ServerLimit set to 120. > 2. httpd seems to not register that it has free workers even when I > set StartServers to 120. There are thousands of idle threads and it > still logs "all workers busy or dying" and kills keepalive > connections. This can be worked around by setting ThreadsPerChild and > ThreadLimit much higher and lowering StartServers/ServerLimit > respectively. > For example with following settings I can easily process over 1500 > concurrent connections without errors and keepalive killing: > ``` > > ThreadsPerChild 200 > ThreadLimit 200 > StartServers 10 > ServerLimit 15 > MinSpareThreads 75 > MaxSpareThreads 3000 > MaxRequestWorkers 3000 > MaxConnectionsPerChild 0 > > ``` > Why does httpd behaves this way? It seems to not be intended > considering ThreadsPerChild is limited to 64 by default by > ThreadLimit. I will try to report a bug to httpd devs as it looks like > one to me. Can you save a few snapshots of the output of /server-status under load? - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
They are all identical VMs. We can also reproduce this on bigger servers. I don't think this is caused by Rocky or Ubuntu config. I can see 2 problems during my tests. 1. httpd does not add any servers when test is running. It kills keepalive connections and logs "all workers busy or dying" but does not add any, maybe one at the end of the test. For example if I set StartServers to 3 it goes up to 4 at the end of the test even though it has ServerLimit set to 120. 2. httpd seems to not register that it has free workers even when I set StartServers to 120. There are thousands of idle threads and it still logs "all workers busy or dying" and kills keepalive connections. This can be worked around by setting ThreadsPerChild and ThreadLimit much higher and lowering StartServers/ServerLimit respectively. For example with following settings I can easily process over 1500 concurrent connections without errors and keepalive killing: ``` ThreadsPerChild 200 ThreadLimit 200 StartServers 10 ServerLimit 15 MinSpareThreads 75 MaxSpareThreads 3000 MaxRequestWorkers 3000 MaxConnectionsPerChild 0 ``` Why does httpd behaves this way? It seems to not be intended considering ThreadsPerChild is limited to 64 by default by ThreadLimit. I will try to report a bug to httpd devs as it looks like one to me. Test results with this config: ``` ab -k -t 900 -c 1500 -n 100 http://rocky/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 10.1.3.11 (be patient) Completed 10 requests Completed 20 requests Completed 30 requests Completed 40 requests Completed 50 requests Completed 60 requests Completed 70 requests Completed 80 requests Completed 90 requests Completed 100 requests Finished 100 requests Server Software:Apache/2.4.37 Server Hostname:10.1.3.11 Server Port:80 Document Path: / Document Length:7620 bytes Concurrency Level: 1500 Time taken for tests: 13.923 seconds Complete requests: 100 Failed requests:0 Write errors: 0 Keep-Alive requests:990813 Total transferred: 7919596479 bytes HTML transferred: 762000 bytes Requests per second:71821.69 [#/sec] (mean) Time per request: 20.885 [ms] (mean) Time per request: 0.014 [ms] (mean, across all concurrent requests) Transfer rate: 555467.60 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 14.1 01057 Processing: 0 20 33.5 161483 Waiting:0 17 20.5 151343 Total: 0 21 36.4 161483 Percentage of the requests served within a certain time (ms) 50% 16 66% 20 75% 23 80% 26 90% 27 95% 31 98% 65 99%221 100% 1483 (longest request) ``` Mateusz Kempski Mateusz Kempski Linux System Administrator XTM International Ltd. Email: mkemp...@xtm-intl.com xtm.cloud On Wed, 24 May 2023 at 05:57, Deepak Goel wrote: > > > > On Tue, May 23, 2023 at 6:19 PM Mateusz Kempski > wrote: >> >> Ubuntu 20 idle: >> ``` >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s >> ``` >> ``` >> top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01 >> Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 >> st >> MiB Mem : 16006.4 total, 15259.6 free,251.0 used,495.8 buff/cache >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem >> ``` >> Ubuntu 20 during test: >> ``` >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s >> ``` >> ``` >> top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16 >> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si, 0.1 >> st >> MiB Mem : 16006.4 total, 14672.6 free,358.1 used,975.7 buff/cache > > > 1. The buff/cache in Ubuntu 20 is almost twice as that of Ubuntu 18. From > what I have read, this should improve the performance of Ubuntu 20 over > Ubuntu 18. However, the reverse is happening. > > 2. The Disk write in Ubuntu 20 is less than that of Ubuntu 18. This again > should improve performance of Ubuntu 20 over Ubuntu 18. Again the reverse is > happening. Are you writing anything to the DISK in your test? > > 3. Is the hardware same for all configurations (cpu, mem, disk iops, network)? > > 4. I wonder if we can open 'Performance Bugs' in Ubuntu & Rocky forums. > > >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15353.9 avail Mem >> >>
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
On Tue, May 23, 2023 at 6:19 PM Mateusz Kempski wrote: > Ubuntu 20 idle: > ``` > Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s > Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s > ``` > ``` > top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01 > Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie > %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > MiB Mem : 16006.4 total, 15259.6 free,251.0 used,495.8 buff/cache > MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem > ``` > Ubuntu 20 during test: > ``` > Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s > Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s > ``` > ``` > top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16 > Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie > %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si, > 0.1 st > MiB Mem : 16006.4 total, 14672.6 free,358.1 used,975.7 buff/cache > 1. The buff/cache in Ubuntu 20 is almost twice as that of Ubuntu 18. From what I have read, this should improve the performance of Ubuntu 20 over Ubuntu 18. However, the reverse is happening. 2. The Disk write in Ubuntu 20 is less than that of Ubuntu 18. This again should improve performance of Ubuntu 20 over Ubuntu 18. Again the reverse is happening. Are you writing anything to the DISK in your test? 3. Is the hardware same for all configurations (cpu, mem, disk iops, network)? 4. I wonder if we can open 'Performance Bugs' in Ubuntu & Rocky forums. > MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15353.9 avail Mem > > ``` > Ubuntu 18 idle: > ``` > Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s > Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > ``` > ``` > top - 12:46:31 up 1 min, 1 user, load average: 0.56, 0.30, 0.11 > Tasks: 226 total, 1 running, 112 sleeping, 0 stopped, 0 zombie > %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > KiB Mem : 16422552 total, 15861664 free, 204228 used, 356660 buff/cache > KiB Swap: 4194300 total, 4194300 free,0 used. 15936820 avail Mem > ``` > Ubuntu 18 during test: > ``` > top - 12:47:07 up 1 min, 1 user, load average: 0.55, 0.31, 0.12 > Tasks: 241 total, 1 running, 127 sleeping, 0 stopped, 0 zombie > %Cpu(s): 11.3 us, 17.0 sy, 0.0 ni, 68.3 id, 0.0 wa, 0.0 hi, 3.3 si, > 0.1 st > KiB Mem : 16422552 total, 15687480 free, 329456 used, 405616 buff/cache > KiB Swap: 4194300 total, 4194300 free,0 used. 15809620 avail Mem > ``` > ``` > Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.76 M/s > Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s > ``` > > > Mateusz Kempski > > On Tue, 23 May 2023 at 12:13, Deepak Goel wrote: > > > > can you please post top, iotop results for both apache 18/20? > > > > On Tue, 23 May 2023, 17:33 Mateusz Kempski, > wrote: > >> > >> On Ubuntu 20 with apache in stock settings + my mpm event config: > >> ``` > >> ab -k -t 900 -c 1000 -n 100 http://ubuntu20/ > >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, > http://www.zeustech.net/ > >> Licensed to The Apache Software Foundation, http://www.apache.org/ > >> > >> Benchmarking 10.1.3.3 (be patient) > >> Completed 10 requests > >> Completed 20 requests > >> Completed 30 requests > >> Completed 40 requests > >> Completed 50 requests > >> Completed 60 requests > >> Completed 70 requests > >> Completed 80 requests > >> Completed 90 requests > >> Completed 100 requests > >> Finished 100 requests > >> > >> > >> Server Software:Apache/2.4.41 > >> Server Hostname:10.1.3.3 > >> Server Port:80 > >> > >> Document Path: / > >> Document Length:10918 bytes > >> > >> Concurrency Level: 1000 > >> Time taken for tests: 19.528 seconds > >> Complete requests: 100 > >> Failed requests:234371 > >> (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032) > >> Write errors: 0 > >> Keep-Alive requests:768663 > >> Total transferred: 8630779988 bytes > >> HTML transferred: 8392262634 bytes > >> Requests per second:51207.31 [#/sec] (mean) > >> Time per request: 19.528 [ms] (mean) > >> Time per request: 0.020 [ms] (mean, across all concurrent > requests) > >> Transfer rate: 431600.59 [Kbytes/sec] received > >> > >> Connection Times (ms) > >> min mean[+/-sd] median max > >> Connect:07 69.0 03022 > >> Processing: 0 12 22.6 92028 > >> Waiting:06 15.8 42023 > >> Total: 0 19 73.8 93038 > >> > >> Percentage of the requests served within a certain
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Ubuntu 20 idle: ``` Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s ``` ``` top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01 Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 16006.4 total, 15259.6 free,251.0 used,495.8 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem ``` Ubuntu 20 during test: ``` Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s ``` ``` top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16 Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si, 0.1 st MiB Mem : 16006.4 total, 14672.6 free,358.1 used,975.7 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15353.9 avail Mem ``` Ubuntu 18 idle: ``` Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s ``` ``` top - 12:46:31 up 1 min, 1 user, load average: 0.56, 0.30, 0.11 Tasks: 226 total, 1 running, 112 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16422552 total, 15861664 free, 204228 used, 356660 buff/cache KiB Swap: 4194300 total, 4194300 free,0 used. 15936820 avail Mem ``` Ubuntu 18 during test: ``` top - 12:47:07 up 1 min, 1 user, load average: 0.55, 0.31, 0.12 Tasks: 241 total, 1 running, 127 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.3 us, 17.0 sy, 0.0 ni, 68.3 id, 0.0 wa, 0.0 hi, 3.3 si, 0.1 st KiB Mem : 16422552 total, 15687480 free, 329456 used, 405616 buff/cache KiB Swap: 4194300 total, 4194300 free,0 used. 15809620 avail Mem ``` ``` Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.76 M/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s ``` Mateusz Kempski On Tue, 23 May 2023 at 12:13, Deepak Goel wrote: > > can you please post top, iotop results for both apache 18/20? > > On Tue, 23 May 2023, 17:33 Mateusz Kempski, > wrote: >> >> On Ubuntu 20 with apache in stock settings + my mpm event config: >> ``` >> ab -k -t 900 -c 1000 -n 100 http://ubuntu20/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.3 (be patient) >> Completed 10 requests >> Completed 20 requests >> Completed 30 requests >> Completed 40 requests >> Completed 50 requests >> Completed 60 requests >> Completed 70 requests >> Completed 80 requests >> Completed 90 requests >> Completed 100 requests >> Finished 100 requests >> >> >> Server Software:Apache/2.4.41 >> Server Hostname:10.1.3.3 >> Server Port:80 >> >> Document Path: / >> Document Length:10918 bytes >> >> Concurrency Level: 1000 >> Time taken for tests: 19.528 seconds >> Complete requests: 100 >> Failed requests:234371 >> (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032) >> Write errors: 0 >> Keep-Alive requests:768663 >> Total transferred: 8630779988 bytes >> HTML transferred: 8392262634 bytes >> Requests per second:51207.31 [#/sec] (mean) >> Time per request: 19.528 [ms] (mean) >> Time per request: 0.020 [ms] (mean, across all concurrent requests) >> Transfer rate: 431600.59 [Kbytes/sec] received >> >> Connection Times (ms) >> min mean[+/-sd] median max >> Connect:07 69.0 03022 >> Processing: 0 12 22.6 92028 >> Waiting:06 15.8 42023 >> Total: 0 19 73.8 93038 >> >> Percentage of the requests served within a certain time (ms) >> 50% 9 >> 66% 13 >> 75% 16 >> 80% 19 >> 90% 30 >> 95% 46 >> 98% 56 >> 99%215 >> 100% 3038 (longest request) >> ``` >> >> On Ubuntu 18 with apache in stock settings + my mpm event config. >> There is ~15 thousands length fails on the first run after apache >> restart then it stays in 100-1k range: >> ``` >> ab -k -t 900 -c 1000 -n 100 http://ubuntu18/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.3 (be patient) >> Completed 10 requests >> Completed 20 requests >> Completed 30 requests >> Completed 40 requests >> Completed 50 requests >> Completed 60 requests >>
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
can you please post top, iotop results for both apache 18/20? On Tue, 23 May 2023, 17:33 Mateusz Kempski, wrote: > On Ubuntu 20 with apache in stock settings + my mpm event config: > ``` > ab -k -t 900 -c 1000 -n 100 http://ubuntu20/ > This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ > Licensed to The Apache Software Foundation, http://www.apache.org/ > > Benchmarking 10.1.3.3 (be patient) > Completed 10 requests > Completed 20 requests > Completed 30 requests > Completed 40 requests > Completed 50 requests > Completed 60 requests > Completed 70 requests > Completed 80 requests > Completed 90 requests > Completed 100 requests > Finished 100 requests > > > Server Software:Apache/2.4.41 > Server Hostname:10.1.3.3 > Server Port:80 > > Document Path: / > Document Length:10918 bytes > > Concurrency Level: 1000 > Time taken for tests: 19.528 seconds > Complete requests: 100 > Failed requests:234371 > (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032) > Write errors: 0 > Keep-Alive requests:768663 > Total transferred: 8630779988 bytes > HTML transferred: 8392262634 bytes > Requests per second:51207.31 [#/sec] (mean) > Time per request: 19.528 [ms] (mean) > Time per request: 0.020 [ms] (mean, across all concurrent requests) > Transfer rate: 431600.59 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect:07 69.0 03022 > Processing: 0 12 22.6 92028 > Waiting:06 15.8 42023 > Total: 0 19 73.8 93038 > > Percentage of the requests served within a certain time (ms) > 50% 9 > 66% 13 > 75% 16 > 80% 19 > 90% 30 > 95% 46 > 98% 56 > 99%215 > 100% 3038 (longest request) > ``` > > On Ubuntu 18 with apache in stock settings + my mpm event config. > There is ~15 thousands length fails on the first run after apache > restart then it stays in 100-1k range: > ``` > ab -k -t 900 -c 1000 -n 100 http://ubuntu18/ > This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ > Licensed to The Apache Software Foundation, http://www.apache.org/ > > Benchmarking 10.1.3.3 (be patient) > Completed 10 requests > Completed 20 requests > Completed 30 requests > Completed 40 requests > Completed 50 requests > Completed 60 requests > Completed 70 requests > Completed 80 requests > Completed 90 requests > Completed 100 requests > Finished 100 requests > > > Server Software:Apache/2.4.29 > Server Hostname:10.1.3.3 > Server Port:80 > > Document Path: / > Document Length:10918 bytes > > Concurrency Level: 1000 > Time taken for tests: 17.221 seconds > Complete requests: 100 > Failed requests:636 > (Connect: 0, Receive: 0, Length: 636, Exceptions: 0) > Write errors: 0 > Keep-Alive requests:990158 > Total transferred: 11220454865 bytes > HTML transferred: 10911056152 bytes > Requests per second:58067.59 [#/sec] (mean) > Time per request: 17.221 [ms] (mean) > Time per request: 0.017 [ms] (mean, across all concurrent requests) > Transfer rate: 636274.18 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect:00 17.4 01020 > Processing: 0 17 11.3 17 377 > Waiting:08 9.0 9 368 > Total: 0 17 21.6 171389 > > Percentage of the requests served within a certain time (ms) > 50% 17 > 66% 18 > 75% 18 > 80% 19 > 90% 20 > 95% 23 > 98% 28 > 99% 36 > 100% 1389 (longest request) > ``` > > Mateusz Kempski > On Mon, 22 May 2023 at 13:05, Deepak Goel wrote: > > > > > > > > On Mon, 22 May 2023, 17:58 Mateusz Kempski, > wrote: > >> > >> @Yann Ylavic: > >> There seems to be no difference in configuration file except for some > comments: > >> ``` > >> diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.rocky > >> 8a > >> # See the httpd.conf(5) man page for more information on this > configuration, > >> # and httpd.service(8) on using and configuring the httpd service. > >> # > >> . > >> ``` > >> I will paste complete Rocky config for completeness at the end of this > message. > >> EnableMMAP and EnableSendfile are set to on, differently to defaults > >> specified in comments. This is default system configuration present on > >> both Rocky 8 and Centos 7. I removed httpd and all config files and > >> installed it again to make sure. > >> ``` > >> # Defaults if commented: EnableMMAP On, EnableSendfile Off > >> # > >> #EnableMMAP off >
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
On Ubuntu 20 with apache in stock settings + my mpm event config: ``` ab -k -t 900 -c 1000 -n 100 http://ubuntu20/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 10.1.3.3 (be patient) Completed 10 requests Completed 20 requests Completed 30 requests Completed 40 requests Completed 50 requests Completed 60 requests Completed 70 requests Completed 80 requests Completed 90 requests Completed 100 requests Finished 100 requests Server Software:Apache/2.4.41 Server Hostname:10.1.3.3 Server Port:80 Document Path: / Document Length:10918 bytes Concurrency Level: 1000 Time taken for tests: 19.528 seconds Complete requests: 100 Failed requests:234371 (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032) Write errors: 0 Keep-Alive requests:768663 Total transferred: 8630779988 bytes HTML transferred: 8392262634 bytes Requests per second:51207.31 [#/sec] (mean) Time per request: 19.528 [ms] (mean) Time per request: 0.020 [ms] (mean, across all concurrent requests) Transfer rate: 431600.59 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:07 69.0 03022 Processing: 0 12 22.6 92028 Waiting:06 15.8 42023 Total: 0 19 73.8 93038 Percentage of the requests served within a certain time (ms) 50% 9 66% 13 75% 16 80% 19 90% 30 95% 46 98% 56 99%215 100% 3038 (longest request) ``` On Ubuntu 18 with apache in stock settings + my mpm event config. There is ~15 thousands length fails on the first run after apache restart then it stays in 100-1k range: ``` ab -k -t 900 -c 1000 -n 100 http://ubuntu18/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 10.1.3.3 (be patient) Completed 10 requests Completed 20 requests Completed 30 requests Completed 40 requests Completed 50 requests Completed 60 requests Completed 70 requests Completed 80 requests Completed 90 requests Completed 100 requests Finished 100 requests Server Software:Apache/2.4.29 Server Hostname:10.1.3.3 Server Port:80 Document Path: / Document Length:10918 bytes Concurrency Level: 1000 Time taken for tests: 17.221 seconds Complete requests: 100 Failed requests:636 (Connect: 0, Receive: 0, Length: 636, Exceptions: 0) Write errors: 0 Keep-Alive requests:990158 Total transferred: 11220454865 bytes HTML transferred: 10911056152 bytes Requests per second:58067.59 [#/sec] (mean) Time per request: 17.221 [ms] (mean) Time per request: 0.017 [ms] (mean, across all concurrent requests) Transfer rate: 636274.18 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 17.4 01020 Processing: 0 17 11.3 17 377 Waiting:08 9.0 9 368 Total: 0 17 21.6 171389 Percentage of the requests served within a certain time (ms) 50% 17 66% 18 75% 18 80% 19 90% 20 95% 23 98% 28 99% 36 100% 1389 (longest request) ``` Mateusz Kempski On Mon, 22 May 2023 at 13:05, Deepak Goel wrote: > > > > On Mon, 22 May 2023, 17:58 Mateusz Kempski, > wrote: >> >> @Yann Ylavic: >> There seems to be no difference in configuration file except for some >> comments: >> ``` >> diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.rocky >> 8a >> # See the httpd.conf(5) man page for more information on this configuration, >> # and httpd.service(8) on using and configuring the httpd service. >> # >> . >> ``` >> I will paste complete Rocky config for completeness at the end of this >> message. >> EnableMMAP and EnableSendfile are set to on, differently to defaults >> specified in comments. This is default system configuration present on >> both Rocky 8 and Centos 7. I removed httpd and all config files and >> installed it again to make sure. >> ``` >> # Defaults if commented: EnableMMAP On, EnableSendfile Off >> # >> #EnableMMAP off >> EnableSendfile on >> ``` >> >> Both systems return: >> ``` >> ulimit -n >> 1024 >> ``` >> There is no custom config for httpd: >> ``` >> stat /etc/systemd/system/httpd.service.d >> stat: cannot stat ‘/etc/systemd/system/httpd.service.d’: No such file >> or directory >> ``` >> systemd itself seems to not apply any limit on both systems: >> ``` >> grep LimitNOFILE /etc/systemd/system.conf >>
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Looks like I did not respond to the right thread so I paste my response below. If you got it twice my apologies. There seems to be no difference in configuration file except for some comments: ``` diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.roc ky 8a # See the httpd.conf(5) man page for more information on this configuration, # and httpd.service(8) on using and configuring the httpd service. # . ``` I will paste complete Rocky config for completeness at the end of this message. EnableMMAP and EnableSendfile are set to on, differently to defaults specified in comments. This is default system configuration present on both Rocky 8 and Centos 7. I removed httpd and all config files and installed it again to make sure. ``` # Defaults if commented: EnableMMAP On, EnableSendfile Off # #EnableMMAP off EnableSendfile on ``` Both systems return: ``` ulimit -n 1024 ``` There is no custom config for httpd: ``` stat /etc/systemd/system/httpd.service.d stat: cannot stat ‘/etc/systemd/system/httpd.service.d’: No such file or directory ``` systemd itself seems to not apply any limit on both systems: ``` grep LimitNOFILE /etc/systemd/system.conf #DefaultLimitNOFILE= ``` After enabling trace there is a lot of messenges about too many open connections and killing keepalive connections. Example snippet: ``` [Mon May 22 11:57:40.146451 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(1808): Too many open connections (73), not accepting new conns in this process [Mon May 22 11:57:40.146455 2023] [mpm_event:trace1] [pid 7710:tid 139800984155904] event.c(1811): Idle workers: 0 [Mon May 22 11:57:40.146457 2023] [mpm_event:trace1] [pid 7710:tid 139800984155904] event.c(1574): All workers are busy or dying, will close 2 keep-alive co nnections [Mon May 22 11:57:40.146526 2023] [mpm_event:debug] [pid 7379:tid 139801403561728] event.c(492): AH00457: Accepting new connections again: 50 active conns ( 27 lingering/0 clogged/0 suspended), 7 idle workers [Mon May 22 11:57:40.146686 2023] [mpm_event:debug] [pid 7511:tid 139800849938176] event.c(1808): Too many open connections (86), not accepting new conns in this process [Mon May 22 11:57:40.146698 2023] [mpm_event:trace1] [pid 7511:tid 139800849938176] event.c(1811): Idle workers: 11 [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7511:tid 139800849938176] event.c(1574): All workers are busy or dying, will close 34 keep-alive c onnections [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7644:tid 139801336452864] event.c(1574): All workers are busy or dying, will close 2 keep-alive co nnections [Mon May 22 11:57:40.146812 2023] [mpm_event:debug] [pid 7511:tid 139800849938176] event.c(492): AH00457: Accepting new connections again: 85 active conns ( 52 lingering/0 clogged/0 suspended), 6 idle workers [Mon May 22 11:57:40.146955 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(492): AH00457: Accepting new connections again: 73 active conns ( 58 lingering/0 clogged/0 suspended), 9 idle workers [Mon May 22 11:57:40.148250 2023] [mpm_event:debug] [pid 7162:tid 139800841545472] event.c(492): AH00457: Accepting new connections again: 48 active conns ( 25 lingering/0 clogged/0 suspended), 2 idle workers [Mon May 22 11:57:40.148562 2023] [mpm_event:debug] [pid 7162:tid 139800841545472] event.c(1808): Too many open connections (48), not accepting new conns in this process [Mon May 22 11:57:40.148573 2023] [mpm_event:trace1] [pid 7162:tid 139800841545472] event.c(1811): Idle workers: 0 [Mon May 22 11:57:40.148921 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(1808): Too many open connections (70), not accepting new conns in this process [Mon May 22 11:57:40.148930 2023] [mpm_event:trace1] [pid 7710:tid 139800984155904] event.c(1811): Idle workers: 1 [Mon May 22 11:57:40.149594 2023] [mpm_event:debug] [pid 7511:tid 139800849938176] event.c(1808): Too many open connections (69), not accepting new conns in this process [Mon May 22 11:57:40.149603 2023] [mpm_event:trace1] [pid 7511:tid 139800849938176] event.c(1811): Idle workers: 1 [Mon May 22 11:57:40.149630 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(492): AH00457: Accepting new connections again: 61 active conns ( 38 lingering/0 clogged/0 suspended), 2 idle workers [Mon May 22 11:57:40.149776 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(1808): Too many open connections (63), not accepting new conns in this process [Mon May 22 11:57:40.149782 2023] [mpm_event:trace1] [pid 7710:tid 139800984155904] event.c(1811): Idle workers: 0 [Mon May 22 11:57:40.149882 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(492): AH00457: Accepting new connections again: 61 active conns ( 38 lingering/0 clogged/0 suspended), 2 idle workers [Mon May 22 11:57:40.149913 2023] [mpm_event:debug] [pid 7511:tid 139800849938176] event.c(492): AH00457: Accepting new connections again: 63 active conns ( 40 lingering/0 clogged/0 suspended), 2
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
On Mon, 22 May 2023, 17:58 Mateusz Kempski, wrote: > @Yann Ylavic: > There seems to be no difference in configuration file except for some > comments: > ``` > diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.rocky > 8a > # See the httpd.conf(5) man page for more information on this > configuration, > # and httpd.service(8) on using and configuring the httpd service. > # > . > ``` > I will paste complete Rocky config for completeness at the end of this > message. > EnableMMAP and EnableSendfile are set to on, differently to defaults > specified in comments. This is default system configuration present on > both Rocky 8 and Centos 7. I removed httpd and all config files and > installed it again to make sure. > ``` > # Defaults if commented: EnableMMAP On, EnableSendfile Off > # > #EnableMMAP off > EnableSendfile on > ``` > > Both systems return: > ``` > ulimit -n > 1024 > ``` > There is no custom config for httpd: > ``` > stat /etc/systemd/system/httpd.service.d > stat: cannot stat ‘/etc/systemd/system/httpd.service.d’: No such file > or directory > ``` > systemd itself seems to not apply any limit on both systems: > ``` > grep LimitNOFILE /etc/systemd/system.conf > #DefaultLimitNOFILE= > ``` > > After enabling trace there is a lot of messenges about too many open > connections and killing keepalive connections. Example snippet: > ``` > [Mon May 22 11:57:40.146451 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(1808): Too many open connections (73), not > accepting new conns in > this process > [Mon May 22 11:57:40.146455 2023] [mpm_event:trace1] [pid 7710:tid > 139800984155904] event.c(1811): Idle workers: 0 > [Mon May 22 11:57:40.146457 2023] [mpm_event:trace1] [pid 7710:tid > 139800984155904] event.c(1574): All workers are busy or dying, will > close 2 keep-alive co > nnections > [Mon May 22 11:57:40.146526 2023] [mpm_event:debug] [pid 7379:tid > 139801403561728] event.c(492): AH00457: Accepting new connections > again: 50 active conns ( > 27 lingering/0 clogged/0 suspended), 7 idle workers > [Mon May 22 11:57:40.146686 2023] [mpm_event:debug] [pid 7511:tid > 139800849938176] event.c(1808): Too many open connections (86), not > accepting new conns in > this process > [Mon May 22 11:57:40.146698 2023] [mpm_event:trace1] [pid 7511:tid > 139800849938176] event.c(1811): Idle workers: 11 > [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7511:tid > 139800849938176] event.c(1574): All workers are busy or dying, will > close 34 keep-alive c > onnections > [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7644:tid > 139801336452864] event.c(1574): All workers are busy or dying, will > close 2 keep-alive co > nnections > [Mon May 22 11:57:40.146812 2023] [mpm_event:debug] [pid 7511:tid > 139800849938176] event.c(492): AH00457: Accepting new connections > again: 85 active conns ( > 52 lingering/0 clogged/0 suspended), 6 idle workers > [Mon May 22 11:57:40.146955 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(492): AH00457: Accepting new connections > again: 73 active conns ( > 58 lingering/0 clogged/0 suspended), 9 idle workers > [Mon May 22 11:57:40.148250 2023] [mpm_event:debug] [pid 7162:tid > 139800841545472] event.c(492): AH00457: Accepting new connections > again: 48 active conns ( > 25 lingering/0 clogged/0 suspended), 2 idle workers > [Mon May 22 11:57:40.148562 2023] [mpm_event:debug] [pid 7162:tid > 139800841545472] event.c(1808): Too many open connections (48), not > accepting new conns in > this process > [Mon May 22 11:57:40.148573 2023] [mpm_event:trace1] [pid 7162:tid > 139800841545472] event.c(1811): Idle workers: 0 > [Mon May 22 11:57:40.148921 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(1808): Too many open connections (70), not > accepting new conns in > this process > [Mon May 22 11:57:40.148930 2023] [mpm_event:trace1] [pid 7710:tid > 139800984155904] event.c(1811): Idle workers: 1 > [Mon May 22 11:57:40.149594 2023] [mpm_event:debug] [pid 7511:tid > 139800849938176] event.c(1808): Too many open connections (69), not > accepting new conns in > this process > [Mon May 22 11:57:40.149603 2023] [mpm_event:trace1] [pid 7511:tid > 139800849938176] event.c(1811): Idle workers: 1 > [Mon May 22 11:57:40.149630 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(492): AH00457: Accepting new connections > again: 61 active conns ( > 38 lingering/0 clogged/0 suspended), 2 idle workers > [Mon May 22 11:57:40.149776 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(1808): Too many open connections (63), not > accepting new conns in > this process > [Mon May 22 11:57:40.149782 2023] [mpm_event:trace1] [pid 7710:tid > 139800984155904] event.c(1811): Idle workers: 0 > [Mon May 22 11:57:40.149882 2023] [mpm_event:debug] [pid 7710:tid > 139800984155904] event.c(492): AH00457: Accepting new connections > again: 61 active conns ( > 38 lingering/0 clogged/0 suspended), 2 idle workers > [Mon May 22
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
@Yann Ylavic: There seems to be no difference in configuration file except for some comments: ``` diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.rocky 8a # See the httpd.conf(5) man page for more information on this configuration, # and httpd.service(8) on using and configuring the httpd service. # . ``` I will paste complete Rocky config for completeness at the end of this message. EnableMMAP and EnableSendfile are set to on, differently to defaults specified in comments. This is default system configuration present on both Rocky 8 and Centos 7. I removed httpd and all config files and installed it again to make sure. ``` # Defaults if commented: EnableMMAP On, EnableSendfile Off # #EnableMMAP off EnableSendfile on ``` Both systems return: ``` ulimit -n 1024 ``` There is no custom config for httpd: ``` stat /etc/systemd/system/httpd.service.d stat: cannot stat ‘/etc/systemd/system/httpd.service.d’: No such file or directory ``` systemd itself seems to not apply any limit on both systems: ``` grep LimitNOFILE /etc/systemd/system.conf #DefaultLimitNOFILE= ``` After enabling trace there is a lot of messenges about too many open connections and killing keepalive connections. Example snippet: ``` [Mon May 22 11:57:40.146451 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(1808): Too many open connections (73), not accepting new conns in this process [Mon May 22 11:57:40.146455 2023] [mpm_event:trace1] [pid 7710:tid 139800984155904] event.c(1811): Idle workers: 0 [Mon May 22 11:57:40.146457 2023] [mpm_event:trace1] [pid 7710:tid 139800984155904] event.c(1574): All workers are busy or dying, will close 2 keep-alive co nnections [Mon May 22 11:57:40.146526 2023] [mpm_event:debug] [pid 7379:tid 139801403561728] event.c(492): AH00457: Accepting new connections again: 50 active conns ( 27 lingering/0 clogged/0 suspended), 7 idle workers [Mon May 22 11:57:40.146686 2023] [mpm_event:debug] [pid 7511:tid 139800849938176] event.c(1808): Too many open connections (86), not accepting new conns in this process [Mon May 22 11:57:40.146698 2023] [mpm_event:trace1] [pid 7511:tid 139800849938176] event.c(1811): Idle workers: 11 [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7511:tid 139800849938176] event.c(1574): All workers are busy or dying, will close 34 keep-alive c onnections [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7644:tid 139801336452864] event.c(1574): All workers are busy or dying, will close 2 keep-alive co nnections [Mon May 22 11:57:40.146812 2023] [mpm_event:debug] [pid 7511:tid 139800849938176] event.c(492): AH00457: Accepting new connections again: 85 active conns ( 52 lingering/0 clogged/0 suspended), 6 idle workers [Mon May 22 11:57:40.146955 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(492): AH00457: Accepting new connections again: 73 active conns ( 58 lingering/0 clogged/0 suspended), 9 idle workers [Mon May 22 11:57:40.148250 2023] [mpm_event:debug] [pid 7162:tid 139800841545472] event.c(492): AH00457: Accepting new connections again: 48 active conns ( 25 lingering/0 clogged/0 suspended), 2 idle workers [Mon May 22 11:57:40.148562 2023] [mpm_event:debug] [pid 7162:tid 139800841545472] event.c(1808): Too many open connections (48), not accepting new conns in this process [Mon May 22 11:57:40.148573 2023] [mpm_event:trace1] [pid 7162:tid 139800841545472] event.c(1811): Idle workers: 0 [Mon May 22 11:57:40.148921 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(1808): Too many open connections (70), not accepting new conns in this process [Mon May 22 11:57:40.148930 2023] [mpm_event:trace1] [pid 7710:tid 139800984155904] event.c(1811): Idle workers: 1 [Mon May 22 11:57:40.149594 2023] [mpm_event:debug] [pid 7511:tid 139800849938176] event.c(1808): Too many open connections (69), not accepting new conns in this process [Mon May 22 11:57:40.149603 2023] [mpm_event:trace1] [pid 7511:tid 139800849938176] event.c(1811): Idle workers: 1 [Mon May 22 11:57:40.149630 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(492): AH00457: Accepting new connections again: 61 active conns ( 38 lingering/0 clogged/0 suspended), 2 idle workers [Mon May 22 11:57:40.149776 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(1808): Too many open connections (63), not accepting new conns in this process [Mon May 22 11:57:40.149782 2023] [mpm_event:trace1] [pid 7710:tid 139800984155904] event.c(1811): Idle workers: 0 [Mon May 22 11:57:40.149882 2023] [mpm_event:debug] [pid 7710:tid 139800984155904] event.c(492): AH00457: Accepting new connections again: 61 active conns ( 38 lingering/0 clogged/0 suspended), 2 idle workers [Mon May 22 11:57:40.149913 2023] [mpm_event:debug] [pid 7511:tid 139800849938176] event.c(492): AH00457: Accepting new connections again: 63 active conns ( 40 lingering/0 clogged/0 suspended), 2 idle workers [Mon May 22 11:57:40.150210 2023] [mpm_event:debug] [pid 7511:tid 139800849938176]
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Hi 1. Please post the test results completely. (Sorry but, saying there is no difference does not help). 2. The memory used in Rocky is comparatively higher than Centos (About 30Meg or so). Also the buff/cache is approx 10 times higher in Rocky than in Centos. 3. Please also post iotop results too. P.S: I hope the top commands is run identical for both Rocky and Centos. Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Mon, May 22, 2023 at 4:28 PM Mateusz Kempski wrote: > I tested again with settings: > ``` > KeepAliveTimeout 300 > MaxKeepAliveRequests 0 > ``` > but there was no difference in results barring normal diffs run-to-run. > > Below is the top of top from both servers when idle and during test. > > Rocky 8 no load: > ``` > top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10 > Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 zombie > %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > MiB Mem : 15824.7 total, 10682.4 free,283.5 used, 4858.8 buff/cache > MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 avail Mem > ``` > Rocky 8 during test: > ``` > top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28 > Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie > %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 5.4 si, > 0.1 st > MiB Mem : 15824.7 total, 9863.0 free,529.3 used, 5432.3 buff/cache > MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 avail Mem > ``` > Centos 7 no load: > ``` > top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00 > Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie > %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, > 0.0 st > KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 buff/cache > KiB Swap:0 total,0 free,0 used. 15720740 avail Mem > ``` > Centos 7 during test: > ``` > top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05 > Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 zombie > %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 3.1 si, > 0.1 st > KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 buff/cache > KiB Swap:0 total,0 free,0 used. 15459544 avail Mem > ``` > > On Mon, 22 May 2023 at 10:34, Deepak Goel wrote: > > > > Hi > > > > I can see about 8000+ requests have timed out in 'Rocky'. This is mostly > due to Apache, which is unable to handle the load. Is it possible to > increase the parameter "KeepAliveTimeout" (and other KeepAlive parameters). > > > > Is it also possible for you to post the hardware utilisations for the 2 > different servers (Centos & Rocky)? > > > > Deepak > > "The greatness of a nation can be judged by the way its animals are > treated - Mahatma Gandhi" > > > > +91 73500 12833 > > deic...@gmail.com > > > > Facebook: https://www.facebook.com/deicool > > LinkedIn: www.linkedin.com/in/deicool > > > > "Plant a Tree, Go Green" > > > > Make In India : http://www.makeinindia.com/home > > > > > > On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski > wrote: > >> > >> Hi all, > >> I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7 > >> install, the other is fresh Rocky 8. I installed httpd (on Centos 7 > >> it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to > >> point to the same static default html file and enabled mpm event on > >> Centos (mpm event is default on Rocky). Then I added following options > >> to default config on both servers: > >> ``` > >> > >> ThreadsPerChild 25 > >> StartServers 3 > >> ServerLimit 120 > >> MinSpareThreads 75 > >> MaxSpareThreads 3000 > >> MaxRequestWorkers 3000 > >> MaxConnectionsPerChild 0 > >> > >> ``` > >> After this is done I performed ab tests with keepalive using different > >> Centos 7 VM in the same local network. Can you help me understand > >> these results? On Centos I am able to complete 1 milion requests at > >> 1000 concurrent connections with little to no errors, however with > >> version 2.4.37 on Rocky I get a lot of failed requests due to length > >> and exceptions. Served content is static so I am assuming this is > >> because keepalive connections are closed by the server. I tried > >> various configurations of KeeAaliveTimeout, MaxKeepAliveRequests, > >> AsyncRequestWorkerFactor and other mpm_event options but I got no > >> better results. The only thing that seems to improve the stability of > >> keepalive connections is setting threads and servers to the moon for > >> example: > >> ``` > >> > >> ThreadsPerChild 50 > >> StartServers 120 > >> ServerLimit 120 > >> MinSpareThreads 6000 > >>
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Hi, On Mon, May 22, 2023 at 12:19 PM Mateusz Kempski wrote: > > Then I added following options > to default config on both servers: > ``` > > ThreadsPerChild 25 > StartServers 3 > ServerLimit 120 > MinSpareThreads 75 > MaxSpareThreads 3000 > MaxRequestWorkers 3000 > MaxConnectionsPerChild 0 > > ``` What is the difference between the two configurations (besides identical MPM parameters)? Things like EnableMMAP and EnableSendfile matter too for instance. Do the two systems have the same `ulimit -n` (or LimitNOFILE in systemd) for httpd? Also, do you see errors in the error_log file? Maybe "LogLevel mpm_event:trace1" could help see what happens while not being too verbose. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
I tested again with settings: ``` KeepAliveTimeout 300 MaxKeepAliveRequests 0 ``` but there was no difference in results barring normal diffs run-to-run. Below is the top of top from both servers when idle and during test. Rocky 8 no load: ``` top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10 Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 15824.7 total, 10682.4 free,283.5 used, 4858.8 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 avail Mem ``` Rocky 8 during test: ``` top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28 Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 5.4 si, 0.1 st MiB Mem : 15824.7 total, 9863.0 free,529.3 used, 5432.3 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 avail Mem ``` Centos 7 no load: ``` top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 buff/cache KiB Swap:0 total,0 free,0 used. 15720740 avail Mem ``` Centos 7 during test: ``` top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05 Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 zombie %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 3.1 si, 0.1 st KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 buff/cache KiB Swap:0 total,0 free,0 used. 15459544 avail Mem ``` On Mon, 22 May 2023 at 10:34, Deepak Goel wrote: > > Hi > > I can see about 8000+ requests have timed out in 'Rocky'. This is mostly due > to Apache, which is unable to handle the load. Is it possible to increase the > parameter "KeepAliveTimeout" (and other KeepAlive parameters). > > Is it also possible for you to post the hardware utilisations for the 2 > different servers (Centos & Rocky)? > > Deepak > "The greatness of a nation can be judged by the way its animals are treated - > Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski > wrote: >> >> Hi all, >> I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7 >> install, the other is fresh Rocky 8. I installed httpd (on Centos 7 >> it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to >> point to the same static default html file and enabled mpm event on >> Centos (mpm event is default on Rocky). Then I added following options >> to default config on both servers: >> ``` >> >> ThreadsPerChild 25 >> StartServers 3 >> ServerLimit 120 >> MinSpareThreads 75 >> MaxSpareThreads 3000 >> MaxRequestWorkers 3000 >> MaxConnectionsPerChild 0 >> >> ``` >> After this is done I performed ab tests with keepalive using different >> Centos 7 VM in the same local network. Can you help me understand >> these results? On Centos I am able to complete 1 milion requests at >> 1000 concurrent connections with little to no errors, however with >> version 2.4.37 on Rocky I get a lot of failed requests due to length >> and exceptions. Served content is static so I am assuming this is >> because keepalive connections are closed by the server. I tried >> various configurations of KeeAaliveTimeout, MaxKeepAliveRequests, >> AsyncRequestWorkerFactor and other mpm_event options but I got no >> better results. The only thing that seems to improve the stability of >> keepalive connections is setting threads and servers to the moon for >> example: >> ``` >> >> ThreadsPerChild 50 >> StartServers 120 >> ServerLimit 120 >> MinSpareThreads 6000 >> MaxSpareThreads 6000 >> MaxRequestWorkers 6000 >> MaxConnectionsPerChild 0 >> >> ``` >> However it stills throws more errors (~8k) than 2.4.6 on the first set >> of settings. This problem occurs only when using keepalive. There are >> no errors when using ab without -k option, although speed is lower. I >> can replicate this issue on newest httpd build from source (2.4.57). >> What is causing this difference of behavior? How can I achieve >> performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing much more >> resources at httpd? These errors seems to be a root cause of a problem >> we have with 502 errors thrown from downstream reverse proxy server >> when httpd kills keepalive connection prematurely and the proxy is >> trying to use this connection. >> >> Below results of ab tests: >> >> Centos 7 VM: >> ``` >> ab -k -t 900 -c 1000 -n 100 http://centos/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >>
Re: [users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Hi I can see about 8000+ requests have timed out in 'Rocky'. This is mostly due to Apache, which is unable to handle the load. Is it possible to increase the parameter "KeepAliveTimeout" (and other KeepAlive parameters). Is it also possible for you to post the hardware utilisations for the 2 different servers (Centos & Rocky)? Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski wrote: > Hi all, > I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7 > install, the other is fresh Rocky 8. I installed httpd (on Centos 7 > it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to > point to the same static default html file and enabled mpm event on > Centos (mpm event is default on Rocky). Then I added following options > to default config on both servers: > ``` > > ThreadsPerChild 25 > StartServers 3 > ServerLimit 120 > MinSpareThreads 75 > MaxSpareThreads 3000 > MaxRequestWorkers 3000 > MaxConnectionsPerChild 0 > > ``` > After this is done I performed ab tests with keepalive using different > Centos 7 VM in the same local network. Can you help me understand > these results? On Centos I am able to complete 1 milion requests at > 1000 concurrent connections with little to no errors, however with > version 2.4.37 on Rocky I get a lot of failed requests due to length > and exceptions. Served content is static so I am assuming this is > because keepalive connections are closed by the server. I tried > various configurations of KeeAaliveTimeout, MaxKeepAliveRequests, > AsyncRequestWorkerFactor and other mpm_event options but I got no > better results. The only thing that seems to improve the stability of > keepalive connections is setting threads and servers to the moon for > example: > ``` > > ThreadsPerChild 50 > StartServers 120 > ServerLimit 120 > MinSpareThreads 6000 > MaxSpareThreads 6000 > MaxRequestWorkers 6000 > MaxConnectionsPerChild 0 > > ``` > However it stills throws more errors (~8k) than 2.4.6 on the first set > of settings. This problem occurs only when using keepalive. There are > no errors when using ab without -k option, although speed is lower. I > can replicate this issue on newest httpd build from source (2.4.57). > What is causing this difference of behavior? How can I achieve > performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing much more > resources at httpd? These errors seems to be a root cause of a problem > we have with 502 errors thrown from downstream reverse proxy server > when httpd kills keepalive connection prematurely and the proxy is > trying to use this connection. > > Below results of ab tests: > > Centos 7 VM: > ``` > ab -k -t 900 -c 1000 -n 100 http://centos/ > This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ > Licensed to The Apache Software Foundation, http://www.apache.org/ > > Benchmarking 10.1.3.3 (be patient) > Completed 10 requests > Completed 20 requests > Completed 30 requests > Completed 40 requests > Completed 50 requests > Completed 60 requests > Completed 70 requests > Completed 80 requests > Completed 90 requests > Completed 100 requests > Finished 100 requests > > > Server Software:Apache/2.4.6 > Server Hostname:10.1.3.3 > Server Port:80 > > Document Path: / > Document Length:7620 bytes > > Concurrency Level: 1000 > Time taken for tests: 15.285 seconds > Complete requests: 100 > Failed requests:67 > (Connect: 0, Receive: 0, Length: 67, Exceptions: 0) > Write errors: 0 > Keep-Alive requests:990567 > Total transferred: 7919057974 bytes > HTML transferred: 7619489460 bytes > Requests per second:65422.95 [#/sec] (mean) > Time per request: 15.285 [ms] (mean) > Time per request: 0.015 [ms] (mean, across all concurrent requests) > Transfer rate: 505945.41 [Kbytes/sec] received > > Connection Times (ms) > min mean[+/-sd] median max > Connect:00 4.7 01042 > Processing: 3 15 16.8 13 467 > Waiting:0 14 16.2 13 433 > Total: 3 15 17.7 131081 > > Percentage of the requests served within a certain time (ms) > 50% 13 > 66% 15 > 75% 16 > 80% 17 > 90% 21 > 95% 23 > 98% 32 > 99% 44 > 100% 1081 (longest request) > ``` > > Rocky 8 VM: > ``` > ab -k -t 900 -c 1000 -n 100 http://rocky/ > This is ApacheBench, Version 2.3 <$Revision: 1430300 $> > Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ > Licensed to The Apache Software Foundation,
[users@httpd] Keepalive closing connections prematurely on high load on newer httpd versions
Hi all, I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7 install, the other is fresh Rocky 8. I installed httpd (on Centos 7 it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to point to the same static default html file and enabled mpm event on Centos (mpm event is default on Rocky). Then I added following options to default config on both servers: ``` ThreadsPerChild 25 StartServers 3 ServerLimit 120 MinSpareThreads 75 MaxSpareThreads 3000 MaxRequestWorkers 3000 MaxConnectionsPerChild 0 ``` After this is done I performed ab tests with keepalive using different Centos 7 VM in the same local network. Can you help me understand these results? On Centos I am able to complete 1 milion requests at 1000 concurrent connections with little to no errors, however with version 2.4.37 on Rocky I get a lot of failed requests due to length and exceptions. Served content is static so I am assuming this is because keepalive connections are closed by the server. I tried various configurations of KeeAaliveTimeout, MaxKeepAliveRequests, AsyncRequestWorkerFactor and other mpm_event options but I got no better results. The only thing that seems to improve the stability of keepalive connections is setting threads and servers to the moon for example: ``` ThreadsPerChild 50 StartServers 120 ServerLimit 120 MinSpareThreads 6000 MaxSpareThreads 6000 MaxRequestWorkers 6000 MaxConnectionsPerChild 0 ``` However it stills throws more errors (~8k) than 2.4.6 on the first set of settings. This problem occurs only when using keepalive. There are no errors when using ab without -k option, although speed is lower. I can replicate this issue on newest httpd build from source (2.4.57). What is causing this difference of behavior? How can I achieve performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing much more resources at httpd? These errors seems to be a root cause of a problem we have with 502 errors thrown from downstream reverse proxy server when httpd kills keepalive connection prematurely and the proxy is trying to use this connection. Below results of ab tests: Centos 7 VM: ``` ab -k -t 900 -c 1000 -n 100 http://centos/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 10.1.3.3 (be patient) Completed 10 requests Completed 20 requests Completed 30 requests Completed 40 requests Completed 50 requests Completed 60 requests Completed 70 requests Completed 80 requests Completed 90 requests Completed 100 requests Finished 100 requests Server Software:Apache/2.4.6 Server Hostname:10.1.3.3 Server Port:80 Document Path: / Document Length:7620 bytes Concurrency Level: 1000 Time taken for tests: 15.285 seconds Complete requests: 100 Failed requests:67 (Connect: 0, Receive: 0, Length: 67, Exceptions: 0) Write errors: 0 Keep-Alive requests:990567 Total transferred: 7919057974 bytes HTML transferred: 7619489460 bytes Requests per second:65422.95 [#/sec] (mean) Time per request: 15.285 [ms] (mean) Time per request: 0.015 [ms] (mean, across all concurrent requests) Transfer rate: 505945.41 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect:00 4.7 01042 Processing: 3 15 16.8 13 467 Waiting:0 14 16.2 13 433 Total: 3 15 17.7 131081 Percentage of the requests served within a certain time (ms) 50% 13 66% 15 75% 16 80% 17 90% 21 95% 23 98% 32 99% 44 100% 1081 (longest request) ``` Rocky 8 VM: ``` ab -k -t 900 -c 1000 -n 100 http://rocky/ This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 10.1.3.11 (be patient) Completed 10 requests Completed 20 requests Completed 30 requests Completed 40 requests Completed 50 requests Completed 60 requests Completed 70 requests Completed 80 requests Completed 90 requests Completed 100 requests Finished 100 requests Server Software:Apache/2.4.37 Server Hostname:10.1.3.11 Server Port:80 Document Path: / Document Length:7620 bytes Concurrency Level: 1000 Time taken for tests: 19.101 seconds Complete requests: 100 Failed requests:93159 (Connect: 0, Receive: 0, Length: 85029, Exceptions: 8130) Write errors: 0 Keep-Alive requests:912753 Total transferred: 7248228337 bytes HTML transferred: 6973694460 bytes Requests per second:52352.12 [#/sec] (mean) Time per request: 19.101 [ms]
