Tested server is Rocky 8. We don't use Ubuntu in prod so I focused on
testing Rocky.
Mateusz Kempski
On Wed, 24 May 2023 at 12:18, Deepak Goel wrote:
>
> The below test results are on which OS? Ubuntu 18 or 20?
>
> On Wed, 24 May 2023, 17:17 Mateusz Kempski,
> wrote:
>>
>> They are all identical VMs. We can also reproduce this on bigger
>> servers. I don't think this is caused by Rocky or Ubuntu config. I can
>> see 2 problems during my tests.
>> 1. httpd does not add any servers when test is running. It kills
>> keepalive connections and logs "all workers busy or dying" but does
>> not add any, maybe one at the end of the test. For example if I set
>> StartServers to 3 it goes up to 4 at the end of the test even though
>> it has ServerLimit set to 120.
>> 2. httpd seems to not register that it has free workers even when I
>> set StartServers to 120. There are thousands of idle threads and it
>> still logs "all workers busy or dying" and kills keepalive
>> connections. This can be worked around by setting ThreadsPerChild and
>> ThreadLimit much higher and lowering StartServers/ServerLimit
>> respectively.
>> For example with following settings I can easily process over 1500
>> concurrent connections without errors and keepalive killing:
>> ```
>>
>> ThreadsPerChild 200
>> ThreadLimit 200
>> StartServers 10
>> ServerLimit 15
>> MinSpareThreads 75
>> MaxSpareThreads 3000
>> MaxRequestWorkers 3000
>> MaxConnectionsPerChild 0
>>
>> ```
>> Why does httpd behaves this way? It seems to not be intended
>> considering ThreadsPerChild is limited to 64 by default by
>> ThreadLimit. I will try to report a bug to httpd devs as it looks like
>> one to me.
>> Test results with this config:
>> ```
>> ab -k -t 900 -c 1500 -n 100 http://rocky/
>> This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
>> Licensed to The Apache Software Foundation, http://www.apache.org/
>>
>> Benchmarking 10.1.3.11 (be patient)
>> Completed 10 requests
>> Completed 20 requests
>> Completed 30 requests
>> Completed 40 requests
>> Completed 50 requests
>> Completed 60 requests
>> Completed 70 requests
>> Completed 80 requests
>> Completed 90 requests
>> Completed 100 requests
>> Finished 100 requests
>>
>>
>> Server Software:Apache/2.4.37
>> Server Hostname:10.1.3.11
>> Server Port:80
>>
>> Document Path: /
>> Document Length:7620 bytes
>>
>> Concurrency Level: 1500
>> Time taken for tests: 13.923 seconds
>> Complete requests: 100
>> Failed requests:0
>> Write errors: 0
>> Keep-Alive requests:990813
>> Total transferred: 7919596479 bytes
>> HTML transferred: 762000 bytes
>> Requests per second:71821.69 [#/sec] (mean)
>> Time per request: 20.885 [ms] (mean)
>> Time per request: 0.014 [ms] (mean, across all concurrent requests)
>> Transfer rate: 555467.60 [Kbytes/sec] received
>>
>> Connection Times (ms)
>> min mean[+/-sd] median max
>> Connect:00 14.1 01057
>> Processing: 0 20 33.5 161483
>> Waiting:0 17 20.5 151343
>> Total: 0 21 36.4 161483
>>
>> Percentage of the requests served within a certain time (ms)
>> 50% 16
>> 66% 20
>> 75% 23
>> 80% 26
>> 90% 27
>> 95% 31
>> 98% 65
>> 99%221
>> 100% 1483 (longest request)
>> ```
>>
>> Mateusz Kempski
>>
>> Mateusz Kempski
>> Linux System Administrator
>> XTM International Ltd.
>> Email: mkemp...@xtm-intl.com
>> xtm.cloud
>>
>>
>>
>>
>> On Wed, 24 May 2023 at 05:57, Deepak Goel wrote:
>> >
>> >
>> >
>> > On Tue, May 23, 2023 at 6:19 PM Mateusz Kempski
>> > wrote:
>> >>
>> >> Ubuntu 20 idle:
>> >> ```
>> >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
>> >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s
>> >> ```
>> >> ```
>> >> top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01
>> >> Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie
>> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si,
>> >> 0.0 st
>> >> MiB Mem : 16006.4 total, 15259.6 free,251.0 used,495.8
>> >> buff/cache
>> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem
>> >> ```
>> >> Ubuntu 20 during test:
>> >> ```
>> >> Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s
>> >> Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s
>> >> ```
>> >> ```
>> >> top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16
>> >> Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie
>> >> %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si,
>> >> 0.1 st
>> >> MiB Mem : 16006.4 total, 14672.6 free,358.1 used,975.7
>> >> buff/cache
