Ubuntu 20 idle: ``` Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s ``` ``` top - 12:31:00 up 2 min, 1 user, load average: 0.13, 0.04, 0.01 Tasks: 239 total, 1 running, 238 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 16006.4 total, 15259.6 free, 251.0 used, 495.8 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15466.5 avail Mem ``` Ubuntu 20 during test: ``` Total DISK READ: 0.00 B/s | Total DISK WRITE: 3.47 M/s Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s ``` ``` top - 12:35:17 up 6 min, 1 user, load average: 6.91, 2.99, 1.16 Tasks: 232 total, 3 running, 229 sleeping, 0 stopped, 0 zombie %Cpu(s): 9.0 us, 21.3 sy, 0.0 ni, 64.4 id, 0.0 wa, 0.0 hi, 5.3 si, 0.1 st MiB Mem : 16006.4 total, 14672.6 free, 358.1 used, 975.7 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15353.9 avail Mem
``` Ubuntu 18 idle: ``` Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s ``` ``` top - 12:46:31 up 1 min, 1 user, load average: 0.56, 0.30, 0.11 Tasks: 226 total, 1 running, 112 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 16422552 total, 15861664 free, 204228 used, 356660 buff/cache KiB Swap: 4194300 total, 4194300 free, 0 used. 15936820 avail Mem ``` Ubuntu 18 during test: ``` top - 12:47:07 up 1 min, 1 user, load average: 0.55, 0.31, 0.12 Tasks: 241 total, 1 running, 127 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.3 us, 17.0 sy, 0.0 ni, 68.3 id, 0.0 wa, 0.0 hi, 3.3 si, 0.1 st KiB Mem : 16422552 total, 15687480 free, 329456 used, 405616 buff/cache KiB Swap: 4194300 total, 4194300 free, 0 used. 15809620 avail Mem ``` ``` Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.76 M/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s ``` Mateusz Kempski On Tue, 23 May 2023 at 12:13, Deepak Goel <deic...@gmail.com> wrote: > > can you please post top, iotop results for both apache 18/20? > > On Tue, 23 May 2023, 17:33 Mateusz Kempski, <mkemp...@xtm-intl.com.invalid> > wrote: >> >> On Ubuntu 20 with apache in stock settings + my mpm event config: >> ``` >> ab -k -t 900 -c 1000 -n 1000000 http://ubuntu20/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.3 (be patient) >> Completed 100000 requests >> Completed 200000 requests >> Completed 300000 requests >> Completed 400000 requests >> Completed 500000 requests >> Completed 600000 requests >> Completed 700000 requests >> Completed 800000 requests >> Completed 900000 requests >> Completed 1000000 requests >> Finished 1000000 requests >> >> >> Server Software: Apache/2.4.41 >> Server Hostname: 10.1.3.3 >> Server Port: 80 >> >> Document Path: / >> Document Length: 10918 bytes >> >> Concurrency Level: 1000 >> Time taken for tests: 19.528 seconds >> Complete requests: 1000000 >> Failed requests: 234371 >> (Connect: 0, Receive: 0, Length: 231339, Exceptions: 3032) >> Write errors: 0 >> Keep-Alive requests: 768663 >> Total transferred: 8630779988 bytes >> HTML transferred: 8392262634 bytes >> Requests per second: 51207.31 [#/sec] (mean) >> Time per request: 19.528 [ms] (mean) >> Time per request: 0.020 [ms] (mean, across all concurrent requests) >> Transfer rate: 431600.59 [Kbytes/sec] received >> >> Connection Times (ms) >> min mean[+/-sd] median max >> Connect: 0 7 69.0 0 3022 >> Processing: 0 12 22.6 9 2028 >> Waiting: 0 6 15.8 4 2023 >> Total: 0 19 73.8 9 3038 >> >> Percentage of the requests served within a certain time (ms) >> 50% 9 >> 66% 13 >> 75% 16 >> 80% 19 >> 90% 30 >> 95% 46 >> 98% 56 >> 99% 215 >> 100% 3038 (longest request) >> ``` >> >> On Ubuntu 18 with apache in stock settings + my mpm event config. >> There is ~15 thousands length fails on the first run after apache >> restart then it stays in 100-1k range: >> ``` >> ab -k -t 900 -c 1000 -n 1000000 http://ubuntu18/ >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> Benchmarking 10.1.3.3 (be patient) >> Completed 100000 requests >> Completed 200000 requests >> Completed 300000 requests >> Completed 400000 requests >> Completed 500000 requests >> Completed 600000 requests >> Completed 700000 requests >> Completed 800000 requests >> Completed 900000 requests >> Completed 1000000 requests >> Finished 1000000 requests >> >> >> Server Software: Apache/2.4.29 >> Server Hostname: 10.1.3.3 >> Server Port: 80 >> >> Document Path: / >> Document Length: 10918 bytes >> >> Concurrency Level: 1000 >> Time taken for tests: 17.221 seconds >> Complete requests: 1000000 >> Failed requests: 636 >> (Connect: 0, Receive: 0, Length: 636, Exceptions: 0) >> Write errors: 0 >> Keep-Alive requests: 990158 >> Total transferred: 11220454865 bytes >> HTML transferred: 10911056152 bytes >> Requests per second: 58067.59 [#/sec] (mean) >> Time per request: 17.221 [ms] (mean) >> Time per request: 0.017 [ms] (mean, across all concurrent requests) >> Transfer rate: 636274.18 [Kbytes/sec] received >> >> Connection Times (ms) >> min mean[+/-sd] median max >> Connect: 0 0 17.4 0 1020 >> Processing: 0 17 11.3 17 377 >> Waiting: 0 8 9.0 9 368 >> Total: 0 17 21.6 17 1389 >> >> Percentage of the requests served within a certain time (ms) >> 50% 17 >> 66% 18 >> 75% 18 >> 80% 19 >> 90% 20 >> 95% 23 >> 98% 28 >> 99% 36 >> 100% 1389 (longest request) >> ``` >> >> Mateusz Kempski >> On Mon, 22 May 2023 at 13:05, Deepak Goel <deic...@gmail.com> wrote: >> > >> > >> > >> > On Mon, 22 May 2023, 17:58 Mateusz Kempski, >> > <mkemp...@xtm-intl.com.invalid> wrote: >> >> >> >> @Yann Ylavic: >> >> There seems to be no difference in configuration file except for some >> >> comments: >> >> ``` >> >> diff -Bbde /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.rocky >> >> 8a >> >> # See the httpd.conf(5) man page for more information on this >> >> configuration, >> >> # and httpd.service(8) on using and configuring the httpd service. >> >> # >> >> . >> >> ``` >> >> I will paste complete Rocky config for completeness at the end of this >> >> message. >> >> EnableMMAP and EnableSendfile are set to on, differently to defaults >> >> specified in comments. This is default system configuration present on >> >> both Rocky 8 and Centos 7. I removed httpd and all config files and >> >> installed it again to make sure. >> >> ``` >> >> # Defaults if commented: EnableMMAP On, EnableSendfile Off >> >> # >> >> #EnableMMAP off >> >> EnableSendfile on >> >> ``` >> >> >> >> Both systems return: >> >> ``` >> >> ulimit -n >> >> 1024 >> >> ``` >> >> There is no custom config for httpd: >> >> ``` >> >> stat /etc/systemd/system/httpd.service.d >> >> stat: cannot stat ‘/etc/systemd/system/httpd.service.d’: No such file >> >> or directory >> >> ``` >> >> systemd itself seems to not apply any limit on both systems: >> >> ``` >> >> grep LimitNOFILE /etc/systemd/system.conf >> >> #DefaultLimitNOFILE= >> >> ``` >> >> >> >> After enabling trace there is a lot of messenges about too many open >> >> connections and killing keepalive connections. Example snippet: >> >> ``` >> >> [Mon May 22 11:57:40.146451 2023] [mpm_event:debug] [pid 7710:tid >> >> 139800984155904] event.c(1808): Too many open connections (73), not >> >> accepting new conns in >> >> this process >> >> [Mon May 22 11:57:40.146455 2023] [mpm_event:trace1] [pid 7710:tid >> >> 139800984155904] event.c(1811): Idle workers: 0 >> >> [Mon May 22 11:57:40.146457 2023] [mpm_event:trace1] [pid 7710:tid >> >> 139800984155904] event.c(1574): All workers are busy or dying, will >> >> close 2 keep-alive co >> >> nnections >> >> [Mon May 22 11:57:40.146526 2023] [mpm_event:debug] [pid 7379:tid >> >> 139801403561728] event.c(492): AH00457: Accepting new connections >> >> again: 50 active conns ( >> >> 27 lingering/0 clogged/0 suspended), 7 idle workers >> >> [Mon May 22 11:57:40.146686 2023] [mpm_event:debug] [pid 7511:tid >> >> 139800849938176] event.c(1808): Too many open connections (86), not >> >> accepting new conns in >> >> this process >> >> [Mon May 22 11:57:40.146698 2023] [mpm_event:trace1] [pid 7511:tid >> >> 139800849938176] event.c(1811): Idle workers: 11 >> >> [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7511:tid >> >> 139800849938176] event.c(1574): All workers are busy or dying, will >> >> close 34 keep-alive c >> >> onnections >> >> [Mon May 22 11:57:40.146701 2023] [mpm_event:trace1] [pid 7644:tid >> >> 139801336452864] event.c(1574): All workers are busy or dying, will >> >> close 2 keep-alive co >> >> nnections >> >> [Mon May 22 11:57:40.146812 2023] [mpm_event:debug] [pid 7511:tid >> >> 139800849938176] event.c(492): AH00457: Accepting new connections >> >> again: 85 active conns ( >> >> 52 lingering/0 clogged/0 suspended), 6 idle workers >> >> [Mon May 22 11:57:40.146955 2023] [mpm_event:debug] [pid 7710:tid >> >> 139800984155904] event.c(492): AH00457: Accepting new connections >> >> again: 73 active conns ( >> >> 58 lingering/0 clogged/0 suspended), 9 idle workers >> >> [Mon May 22 11:57:40.148250 2023] [mpm_event:debug] [pid 7162:tid >> >> 139800841545472] event.c(492): AH00457: Accepting new connections >> >> again: 48 active conns ( >> >> 25 lingering/0 clogged/0 suspended), 2 idle workers >> >> [Mon May 22 11:57:40.148562 2023] [mpm_event:debug] [pid 7162:tid >> >> 139800841545472] event.c(1808): Too many open connections (48), not >> >> accepting new conns in >> >> this process >> >> [Mon May 22 11:57:40.148573 2023] [mpm_event:trace1] [pid 7162:tid >> >> 139800841545472] event.c(1811): Idle workers: 0 >> >> [Mon May 22 11:57:40.148921 2023] [mpm_event:debug] [pid 7710:tid >> >> 139800984155904] event.c(1808): Too many open connections (70), not >> >> accepting new conns in >> >> this process >> >> [Mon May 22 11:57:40.148930 2023] [mpm_event:trace1] [pid 7710:tid >> >> 139800984155904] event.c(1811): Idle workers: 1 >> >> [Mon May 22 11:57:40.149594 2023] [mpm_event:debug] [pid 7511:tid >> >> 139800849938176] event.c(1808): Too many open connections (69), not >> >> accepting new conns in >> >> this process >> >> [Mon May 22 11:57:40.149603 2023] [mpm_event:trace1] [pid 7511:tid >> >> 139800849938176] event.c(1811): Idle workers: 1 >> >> [Mon May 22 11:57:40.149630 2023] [mpm_event:debug] [pid 7710:tid >> >> 139800984155904] event.c(492): AH00457: Accepting new connections >> >> again: 61 active conns ( >> >> 38 lingering/0 clogged/0 suspended), 2 idle workers >> >> [Mon May 22 11:57:40.149776 2023] [mpm_event:debug] [pid 7710:tid >> >> 139800984155904] event.c(1808): Too many open connections (63), not >> >> accepting new conns in >> >> this process >> >> [Mon May 22 11:57:40.149782 2023] [mpm_event:trace1] [pid 7710:tid >> >> 139800984155904] event.c(1811): Idle workers: 0 >> >> [Mon May 22 11:57:40.149882 2023] [mpm_event:debug] [pid 7710:tid >> >> 139800984155904] event.c(492): AH00457: Accepting new connections >> >> again: 61 active conns ( >> >> 38 lingering/0 clogged/0 suspended), 2 idle workers >> >> [Mon May 22 11:57:40.149913 2023] [mpm_event:debug] [pid 7511:tid >> >> 139800849938176] event.c(492): AH00457: Accepting new connections >> >> again: 63 active conns ( >> >> 40 lingering/0 clogged/0 suspended), 2 idle workers >> >> [Mon May 22 11:57:40.150210 2023] [mpm_event:debug] [pid 7511:tid >> >> 139800849938176] event.c(1808): Too many open connections (62), not >> >> accepting new conns in >> >> this process >> >> [Mon May 22 11:57:40.150216 2023] [mpm_event:trace1] [pid 7511:tid >> >> 139800849938176] event.c(1811): Idle workers: 0 >> >> ``` >> >> >> >> @Deepak Goel: >> >> 1. Here are result with previously mentioned KeepAlive parameters: >> >> Rocky 8: >> >> ``` >> >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/ >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> >> >> Benchmarking 10.1.3.11 (be patient) >> >> Completed 100000 requests >> >> Completed 200000 requests >> >> Completed 300000 requests >> >> Completed 400000 requests >> >> Completed 500000 requests >> >> Completed 600000 requests >> >> Completed 700000 requests >> >> Completed 800000 requests >> >> Completed 900000 requests >> >> Completed 1000000 requests >> >> Finished 1000000 requests >> >> >> >> >> >> Server Software: Apache/2.4.37 >> >> Server Hostname: 10.1.3.11 >> >> Server Port: 80 >> >> >> >> Document Path: / >> >> Document Length: 7620 bytes >> >> >> >> Concurrency Level: 1000 >> >> Time taken for tests: 17.287 seconds >> >> Complete requests: 1000000 >> >> Failed requests: 98218 >> >> (Connect: 0, Receive: 0, Length: 88397, Exceptions: 9821) >> >> Write errors: 0 >> >> Keep-Alive requests: 911603 >> >> Total transferred: 7214426142 bytes >> >> HTML transferred: 6946414860 bytes >> >> Requests per second: 57845.81 [#/sec] (mean) >> >> Time per request: 17.287 [ms] (mean) >> >> Time per request: 0.017 [ms] (mean, across all concurrent requests) >> >> Transfer rate: 407543.30 [Kbytes/sec] received >> >> >> >> Connection Times (ms) >> >> min mean[+/-sd] median max >> >> Connect: 0 4 59.9 0 3015 >> >> Processing: 0 13 14.5 13 432 >> >> Waiting: 0 13 14.1 13 432 >> >> Total: 0 17 63.2 15 3036 >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> 50% 15 >> >> 66% 17 >> >> 75% 17 >> >> 80% 18 >> >> 90% 21 >> >> 95% 23 >> >> 98% 28 >> >> 99% 36 >> >> 100% 3036 (longest request) >> >> ``` >> >> Centos 7: >> >> ``` >> >> ab -k -t 900 -c 1000 -n 1000000 http://10.1.3.3/ >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> >> >> Benchmarking 10.1.3.3 (be patient) >> >> Completed 100000 requests >> >> Completed 200000 requests >> >> Completed 300000 requests >> >> Completed 400000 requests >> >> Completed 500000 requests >> >> Completed 600000 requests >> >> Completed 700000 requests >> >> Completed 800000 requests >> >> Completed 900000 requests >> >> Completed 1000000 requests >> >> Finished 1000000 requests >> >> >> >> >> >> Server Software: Apache/2.4.6 >> >> Server Hostname: 10.1.3.3 >> >> Server Port: 80 >> >> >> >> Document Path: / >> >> Document Length: 7620 bytes >> >> >> >> Concurrency Level: 1000 >> >> Time taken for tests: 20.731 seconds >> >> Complete requests: 1000000 >> >> Failed requests: 14 >> >> (Connect: 0, Receive: 0, Length: 14, Exceptions: 0) >> >> Write errors: 0 >> >> Keep-Alive requests: 999986 >> >> Total transferred: 7913889204 bytes >> >> HTML transferred: 7619893320 bytes >> >> Requests per second: 48236.94 [#/sec] (mean) >> >> Time per request: 20.731 [ms] (mean) >> >> Time per request: 0.021 [ms] (mean, across all concurrent requests) >> >> Transfer rate: 372794.72 [Kbytes/sec] received >> >> >> >> Connection Times (ms) >> >> min mean[+/-sd] median max >> >> Connect: 0 0 13.1 0 1025 >> >> Processing: 2 21 7.9 20 821 >> >> Waiting: 0 20 7.8 20 821 >> >> Total: 2 21 15.7 20 1244 >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> 50% 20 >> >> 66% 22 >> >> 75% 23 >> >> 80% 24 >> >> 90% 25 >> >> 95% 27 >> >> 98% 29 >> >> 99% 30 >> >> 100% 1244 (longest request) >> >> ``` >> >> 3. iotop results: >> >> Centos 7 idle: >> >> ``` >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> ``` >> >> Rocky 8 idle: >> >> ``` >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> ``` >> >> Centos 7 during test: >> >> ``` >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 4.56 M/s >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> ``` >> >> Rocky 8 during test: >> >> ``` >> >> Total DISK READ : 0.00 B/s | Total DISK WRITE : 5.35 M/s >> > >> > >> > The DISK WRITE is higher in Rocky as compared to Centos. >> > >> > Is it possible for you to test on Ubuntu 18/20 as well? ( Seems Rocky >> > algo's are taking a performance hit ) >> >> >> >> Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s >> >> ``` >> >> I ran top command without parameters on both systems - looks like >> >> Centos defaults to KiB and Rocky to MiB. >> >> >> >> Complete httpd.conf: >> >> ``` >> >> # >> >> # This is the main Apache HTTP server configuration file. It contains the >> >> # configuration directives that give the server its instructions. >> >> # See <URL:http://httpd.apache.org/docs/2.4/> for detailed information. >> >> # In particular, see >> >> # <URL:http://httpd.apache.org/docs/2.4/mod/directives.html> >> >> # for a discussion of each configuration directive. >> >> # >> >> # See the httpd.conf(5) man page for more information on this >> >> configuration, >> >> # and httpd.service(8) on using and configuring the httpd service. >> >> # >> >> # Do NOT simply read the instructions in here without understanding >> >> # what they do. They're here only as hints or reminders. If you are >> >> unsure >> >> # consult the online docs. You have been warned. >> >> # >> >> # Configuration and logfile names: If the filenames you specify for many >> >> # of the server's control files begin with "/" (or "drive:/" for Win32), >> >> the >> >> # server will use that explicit path. If the filenames do *not* begin >> >> # with "/", the value of ServerRoot is prepended -- so 'log/access_log' >> >> # with ServerRoot set to '/www' will be interpreted by the >> >> # server as '/www/log/access_log', where as '/log/access_log' will be >> >> # interpreted as '/log/access_log'. >> >> >> >> # >> >> # ServerRoot: The top of the directory tree under which the server's >> >> # configuration, error, and log files are kept. >> >> # >> >> # Do not add a slash at the end of the directory path. If you point >> >> # ServerRoot at a non-local disk, be sure to specify a local disk on the >> >> # Mutex directive, if file-based mutexes are used. If you wish to share >> >> the >> >> # same ServerRoot for multiple httpd daemons, you will need to change at >> >> # least PidFile. >> >> # >> >> ServerRoot "/etc/httpd" >> >> >> >> # >> >> # Listen: Allows you to bind Apache to specific IP addresses and/or >> >> # ports, instead of the default. See also the <VirtualHost> >> >> # directive. >> >> # >> >> # Change this to Listen on specific IP addresses as shown below to >> >> # prevent Apache from glomming onto all bound IP addresses. >> >> # >> >> #Listen 12.34.56.78:80 >> >> Listen 80 >> >> >> >> # >> >> # Dynamic Shared Object (DSO) Support >> >> # >> >> # To be able to use the functionality of a module which was built as a >> >> DSO you >> >> # have to place corresponding `LoadModule' lines at this location so the >> >> # directives contained in it are actually available _before_ they are >> >> used. >> >> # Statically compiled modules (those listed by `httpd -l') do not need >> >> # to be loaded here. >> >> # >> >> # Example: >> >> # LoadModule foo_module modules/mod_foo.so >> >> # >> >> Include conf.modules.d/*.conf >> >> >> >> # >> >> # If you wish httpd to run as a different user or group, you must run >> >> # httpd as root initially and it will switch. >> >> # >> >> # User/Group: The name (or #number) of the user/group to run httpd as. >> >> # It is usually good practice to create a dedicated user and group for >> >> # running httpd, as with most system services. >> >> # >> >> User apache >> >> Group apache >> >> >> >> # 'Main' server configuration >> >> # >> >> # The directives in this section set up the values used by the 'main' >> >> # server, which responds to any requests that aren't handled by a >> >> # <VirtualHost> definition. These values also provide defaults for >> >> # any <VirtualHost> containers you may define later in the file. >> >> # >> >> # All of these directives may appear inside <VirtualHost> containers, >> >> # in which case these default settings will be overridden for the >> >> # virtual host being defined. >> >> # >> >> >> >> # >> >> # ServerAdmin: Your address, where problems with the server should be >> >> # e-mailed. This address appears on some server-generated pages, such >> >> # as error documents. e.g. ad...@your-domain.com >> >> # >> >> ServerAdmin root@localhost >> >> >> >> # >> >> # ServerName gives the name and port that the server uses to identify >> >> itself. >> >> # This can often be determined automatically, but we recommend you specify >> >> # it explicitly to prevent problems during startup. >> >> # >> >> # If your host doesn't have a registered DNS name, enter its IP address >> >> here. >> >> # >> >> #ServerName www.example.com:80 >> >> >> >> # >> >> # Deny access to the entirety of your server's filesystem. You must >> >> # explicitly permit access to web content directories in other >> >> # <Directory> blocks below. >> >> # >> >> <Directory /> >> >> AllowOverride none >> >> Require all denied >> >> </Directory> >> >> >> >> # >> >> # Note that from this point forward you must specifically allow >> >> # particular features to be enabled - so if something's not working as >> >> # you might expect, make sure that you have specifically enabled it >> >> # below. >> >> # >> >> >> >> # >> >> # DocumentRoot: The directory out of which you will serve your >> >> # documents. By default, all requests are taken from this directory, but >> >> # symbolic links and aliases may be used to point to other locations. >> >> # >> >> DocumentRoot "/var/www/html" >> >> >> >> # >> >> # Relax access to content within /var/www. >> >> # >> >> <Directory "/var/www"> >> >> AllowOverride None >> >> # Allow open access: >> >> Require all granted >> >> </Directory> >> >> >> >> # Further relax access to the default document root: >> >> <Directory "/var/www/html"> >> >> # >> >> # Possible values for the Options directive are "None", "All", >> >> # or any combination of: >> >> # Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI >> >> MultiViews >> >> # >> >> # Note that "MultiViews" must be named *explicitly* --- "Options All" >> >> # doesn't give it to you. >> >> # >> >> # The Options directive is both complicated and important. Please see >> >> # http://httpd.apache.org/docs/2.4/mod/core.html#options >> >> # for more information. >> >> # >> >> Options Indexes FollowSymLinks >> >> >> >> # >> >> # AllowOverride controls what directives may be placed in .htaccess >> >> files. >> >> # It can be "All", "None", or any combination of the keywords: >> >> # Options FileInfo AuthConfig Limit >> >> # >> >> AllowOverride None >> >> >> >> # >> >> # Controls who can get stuff from this server. >> >> # >> >> Require all granted >> >> </Directory> >> >> >> >> # >> >> # DirectoryIndex: sets the file that Apache will serve if a directory >> >> # is requested. >> >> # >> >> <IfModule dir_module> >> >> DirectoryIndex index.html >> >> </IfModule> >> >> >> >> # >> >> # The following lines prevent .htaccess and .htpasswd files from being >> >> # viewed by Web clients. >> >> # >> >> <Files ".ht*"> >> >> Require all denied >> >> </Files> >> >> >> >> # >> >> # ErrorLog: The location of the error log file. >> >> # If you do not specify an ErrorLog directive within a <VirtualHost> >> >> # container, error messages relating to that virtual host will be >> >> # logged here. If you *do* define an error logfile for a <VirtualHost> >> >> # container, that host's errors will be logged there and not here. >> >> # >> >> ErrorLog "logs/error_log" >> >> >> >> # >> >> # LogLevel: Control the number of messages logged to the error_log. >> >> # Possible values include: debug, info, notice, warn, error, crit, >> >> # alert, emerg. >> >> # >> >> LogLevel warn >> >> >> >> <IfModule log_config_module> >> >> # >> >> # The following directives define some format nicknames for use with >> >> # a CustomLog directive (see below). >> >> # >> >> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" >> >> \"%{User-Agent}i\"" combined >> >> LogFormat "%h %l %u %t \"%r\" %>s %b" common >> >> >> >> <IfModule logio_module> >> >> # You need to enable mod_logio.c to use %I and %O >> >> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" >> >> \"%{User-Agent}i\" %I %O" combinedio >> >> </IfModule> >> >> >> >> # >> >> # The location and format of the access logfile (Common Logfile >> >> Format). >> >> # If you do not define any access logfiles within a <VirtualHost> >> >> # container, they will be logged here. Contrariwise, if you *do* >> >> # define per-<VirtualHost> access logfiles, transactions will be >> >> # logged therein and *not* in this file. >> >> # >> >> #CustomLog "logs/access_log" common >> >> >> >> # >> >> # If you prefer a logfile with access, agent, and referer information >> >> # (Combined Logfile Format) you can use the following directive. >> >> # >> >> CustomLog "logs/access_log" combined >> >> </IfModule> >> >> >> >> <IfModule alias_module> >> >> # >> >> # Redirect: Allows you to tell clients about documents that used to >> >> # exist in your server's namespace, but do not anymore. The client >> >> # will make a new request for the document at its new location. >> >> # Example: >> >> # Redirect permanent /foo http://www.example.com/bar >> >> >> >> # >> >> # Alias: Maps web paths into filesystem paths and is used to >> >> # access content that does not live under the DocumentRoot. >> >> # Example: >> >> # Alias /webpath /full/filesystem/path >> >> # >> >> # If you include a trailing / on /webpath then the server will >> >> # require it to be present in the URL. You will also likely >> >> # need to provide a <Directory> section to allow access to >> >> # the filesystem path. >> >> >> >> # >> >> # ScriptAlias: This controls which directories contain server scripts. >> >> # ScriptAliases are essentially the same as Aliases, except that >> >> # documents in the target directory are treated as applications and >> >> # run by the server when requested rather than as documents sent to the >> >> # client. The same rules about trailing "/" apply to ScriptAlias >> >> # directives as to Alias. >> >> # >> >> ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" >> >> >> >> </IfModule> >> >> >> >> # >> >> # "/var/www/cgi-bin" should be changed to whatever your ScriptAliased >> >> # CGI directory exists, if you have that configured. >> >> # >> >> <Directory "/var/www/cgi-bin"> >> >> AllowOverride None >> >> Options None >> >> Require all granted >> >> </Directory> >> >> >> >> <IfModule mime_module> >> >> # >> >> # TypesConfig points to the file containing the list of mappings from >> >> # filename extension to MIME-type. >> >> # >> >> TypesConfig /etc/mime.types >> >> >> >> # >> >> # AddType allows you to add to or override the MIME configuration >> >> # file specified in TypesConfig for specific file types. >> >> # >> >> #AddType application/x-gzip .tgz >> >> # >> >> # AddEncoding allows you to have certain browsers uncompress >> >> # information on the fly. Note: Not all browsers support this. >> >> # >> >> #AddEncoding x-compress .Z >> >> #AddEncoding x-gzip .gz .tgz >> >> # >> >> # If the AddEncoding directives above are commented-out, then you >> >> # probably should define those extensions to indicate media types: >> >> # >> >> AddType application/x-compress .Z >> >> AddType application/x-gzip .gz .tgz >> >> >> >> # >> >> # AddHandler allows you to map certain file extensions to "handlers": >> >> # actions unrelated to filetype. These can be either built into the >> >> server >> >> # or added with the Action directive (see below) >> >> # >> >> # To use CGI scripts outside of ScriptAliased directories: >> >> # (You will also need to add "ExecCGI" to the "Options" directive.) >> >> # >> >> #AddHandler cgi-script .cgi >> >> >> >> # For type maps (negotiated resources): >> >> #AddHandler type-map var >> >> >> >> # >> >> # Filters allow you to process content before it is sent to the client. >> >> # >> >> # To parse .shtml files for server-side includes (SSI): >> >> # (You will also need to add "Includes" to the "Options" directive.) >> >> # >> >> AddType text/html .shtml >> >> AddOutputFilter INCLUDES .shtml >> >> </IfModule> >> >> >> >> # >> >> # Specify a default charset for all content served; this enables >> >> # interpretation of all content as UTF-8 by default. To use the >> >> # default browser choice (ISO-8859-1), or to allow the META tags >> >> # in HTML content to override this choice, comment out this >> >> # directive: >> >> # >> >> AddDefaultCharset UTF-8 >> >> >> >> <IfModule mime_magic_module> >> >> # >> >> # The mod_mime_magic module allows the server to use various hints >> >> from the >> >> # contents of the file itself to determine its type. The MIMEMagicFile >> >> # directive tells the module where the hint definitions are located. >> >> # >> >> MIMEMagicFile conf/magic >> >> </IfModule> >> >> >> >> # >> >> # Customizable error responses come in three flavors: >> >> # 1) plain text 2) local redirects 3) external redirects >> >> # >> >> # Some examples: >> >> #ErrorDocument 500 "The server made a boo boo." >> >> #ErrorDocument 404 /missing.html >> >> #ErrorDocument 404 "/cgi-bin/missing_handler.pl" >> >> #ErrorDocument 402 http://www.example.com/subscription_info.html >> >> # >> >> >> >> # >> >> # EnableMMAP and EnableSendfile: On systems that support it, >> >> # memory-mapping or the sendfile syscall may be used to deliver >> >> # files. This usually improves server performance, but must >> >> # be turned off when serving from networked-mounted >> >> # filesystems or if support for these functions is otherwise >> >> # broken on your system. >> >> # Defaults if commented: EnableMMAP On, EnableSendfile Off >> >> # >> >> #EnableMMAP off >> >> EnableSendfile on >> >> >> >> # Supplemental configuration >> >> # >> >> # Load config files in the "/etc/httpd/conf.d" directory, if any. >> >> IncludeOptional conf.d/*.conf >> >> >> >> <IfModule mpm_event_module> >> >> ThreadsPerChild 25 >> >> StartServers 3 >> >> ServerLimit 120 >> >> MinSpareThreads 75 >> >> MaxSpareThreads 3000 >> >> MaxRequestWorkers 3000 >> >> MaxConnectionsPerChild 0 >> >> </IfModule> >> >> ``` >> >> >> >> Mateusz Kempski >> >> Linux System Administrator >> >> XTM International Ltd. >> >> Email: mkemp...@xtm-intl.com >> >> xtm.cloud >> >> >> >> >> >> >> >> >> >> On Mon, 22 May 2023 at 11:16, Deepak Goel <deic...@gmail.com> wrote: >> >> > >> >> > Hi >> >> > >> >> > 1. Please post the test results completely. (Sorry but, saying there is >> >> > no difference does not help). >> >> > 2. The memory used in Rocky is comparatively higher than Centos (About >> >> > 30Meg or so). Also the buff/cache is approx 10 times higher in Rocky >> >> > than in Centos. >> >> > 3. Please also post iotop results too. >> >> > >> >> > P.S: I hope the top commands is run identical for both Rocky and Centos. >> >> > >> >> > Deepak >> >> > "The greatness of a nation can be judged by the way its animals are >> >> > treated - Mahatma Gandhi" >> >> > >> >> > +91 73500 12833 >> >> > deic...@gmail.com >> >> > >> >> > Facebook: https://www.facebook.com/deicool >> >> > LinkedIn: www.linkedin.com/in/deicool >> >> > >> >> > "Plant a Tree, Go Green" >> >> > >> >> > Make In India : http://www.makeinindia.com/home >> >> > >> >> > >> >> > >> >> > On Mon, May 22, 2023 at 4:28 PM Mateusz Kempski >> >> > <mkemp...@xtm-intl.com.invalid> wrote: >> >> >> >> >> >> I tested again with settings: >> >> >> ``` >> >> >> KeepAliveTimeout 300 >> >> >> MaxKeepAliveRequests 0 >> >> >> ``` >> >> >> but there was no difference in results barring normal diffs run-to-run. >> >> >> >> >> >> Below is the top of top from both servers when idle and during test. >> >> >> >> >> >> Rocky 8 no load: >> >> >> ``` >> >> >> top - 10:49:23 up 4 min, 1 user, load average: 3.27, 2.56, 1.10 >> >> >> Tasks: 254 total, 2 running, 252 sleeping, 0 stopped, 0 zombie >> >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 >> >> >> si, 0.0 st >> >> >> MiB Mem : 15824.7 total, 10682.4 free, 283.5 used, 4858.8 >> >> >> buff/cache >> >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15259.3 avail >> >> >> Mem >> >> >> ``` >> >> >> Rocky 8 during test: >> >> >> ``` >> >> >> top - 10:50:29 up 5 min, 1 user, load average: 4.33, 2.80, 1.28 >> >> >> Tasks: 232 total, 2 running, 230 sleeping, 0 stopped, 0 zombie >> >> >> %Cpu(s): 13.7 us, 16.9 sy, 0.0 ni, 63.9 id, 0.0 wa, 0.0 hi, 5.4 >> >> >> si, 0.1 st >> >> >> MiB Mem : 15824.7 total, 9863.0 free, 529.3 used, 5432.3 >> >> >> buff/cache >> >> >> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 15012.2 avail >> >> >> Mem >> >> >> ``` >> >> >> Centos 7 no load: >> >> >> ``` >> >> >> top - 10:52:17 up 0 min, 1 user, load average: 0.00, 0.00, 0.00 >> >> >> Tasks: 201 total, 1 running, 200 sleeping, 0 stopped, 0 zombie >> >> >> %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 >> >> >> si, 0.0 st >> >> >> KiB Mem : 16264300 total, 15831896 free, 297124 used, 135280 >> >> >> buff/cache >> >> >> KiB Swap: 0 total, 0 free, 0 used. 15720740 avail >> >> >> Mem >> >> >> ``` >> >> >> Centos 7 during test: >> >> >> ``` >> >> >> top - 10:53:21 up 1 min, 1 user, load average: 0.62, 0.16, 0.05 >> >> >> Tasks: 218 total, 3 running, 215 sleeping, 0 stopped, 0 zombie >> >> >> %Cpu(s): 17.6 us, 18.9 sy, 0.0 ni, 60.4 id, 0.0 wa, 0.0 hi, 3.1 >> >> >> si, 0.1 st >> >> >> KiB Mem : 16264300 total, 14973128 free, 503104 used, 788068 >> >> >> buff/cache >> >> >> KiB Swap: 0 total, 0 free, 0 used. 15459544 avail >> >> >> Mem >> >> >> ``` >> >> >> >> >> >> On Mon, 22 May 2023 at 10:34, Deepak Goel <deic...@gmail.com> wrote: >> >> >> > >> >> >> > Hi >> >> >> > >> >> >> > I can see about 8000+ requests have timed out in 'Rocky'. This is >> >> >> > mostly due to Apache, which is unable to handle the load. Is it >> >> >> > possible to increase the parameter "KeepAliveTimeout" (and other >> >> >> > KeepAlive parameters). >> >> >> > >> >> >> > Is it also possible for you to post the hardware utilisations for >> >> >> > the 2 different servers (Centos & Rocky)? >> >> >> > >> >> >> > Deepak >> >> >> > "The greatness of a nation can be judged by the way its animals are >> >> >> > treated - Mahatma Gandhi" >> >> >> > >> >> >> > +91 73500 12833 >> >> >> > deic...@gmail.com >> >> >> > >> >> >> > Facebook: https://www.facebook.com/deicool >> >> >> > LinkedIn: www.linkedin.com/in/deicool >> >> >> > >> >> >> > "Plant a Tree, Go Green" >> >> >> > >> >> >> > Make In India : http://www.makeinindia.com/home >> >> >> > >> >> >> > >> >> >> > On Mon, May 22, 2023 at 3:49 PM Mateusz Kempski >> >> >> > <mkemp...@xtm-intl.com.invalid> wrote: >> >> >> >> >> >> >> >> Hi all, >> >> >> >> I have two identical VMs - 16GB RAM, 16 vCPUs. One is fresh Centos 7 >> >> >> >> install, the other is fresh Rocky 8. I installed httpd (on Centos 7 >> >> >> >> it's version 2.4.6 and on Rocky 8 it's 2.4.37), configured them to >> >> >> >> point to the same static default html file and enabled mpm event on >> >> >> >> Centos (mpm event is default on Rocky). Then I added following >> >> >> >> options >> >> >> >> to default config on both servers: >> >> >> >> ``` >> >> >> >> <IfModule mpm_event_module> >> >> >> >> ThreadsPerChild 25 >> >> >> >> StartServers 3 >> >> >> >> ServerLimit 120 >> >> >> >> MinSpareThreads 75 >> >> >> >> MaxSpareThreads 3000 >> >> >> >> MaxRequestWorkers 3000 >> >> >> >> MaxConnectionsPerChild 0 >> >> >> >> </IfModule> >> >> >> >> ``` >> >> >> >> After this is done I performed ab tests with keepalive using >> >> >> >> different >> >> >> >> Centos 7 VM in the same local network. Can you help me understand >> >> >> >> these results? On Centos I am able to complete 1 milion requests at >> >> >> >> 1000 concurrent connections with little to no errors, however with >> >> >> >> version 2.4.37 on Rocky I get a lot of failed requests due to length >> >> >> >> and exceptions. Served content is static so I am assuming this is >> >> >> >> because keepalive connections are closed by the server. I tried >> >> >> >> various configurations of KeeAaliveTimeout, MaxKeepAliveRequests, >> >> >> >> AsyncRequestWorkerFactor and other mpm_event options but I got no >> >> >> >> better results. The only thing that seems to improve the stability >> >> >> >> of >> >> >> >> keepalive connections is setting threads and servers to the moon for >> >> >> >> example: >> >> >> >> ``` >> >> >> >> <IfModule mpm_event_module> >> >> >> >> ThreadsPerChild 50 >> >> >> >> StartServers 120 >> >> >> >> ServerLimit 120 >> >> >> >> MinSpareThreads 6000 >> >> >> >> MaxSpareThreads 6000 >> >> >> >> MaxRequestWorkers 6000 >> >> >> >> MaxConnectionsPerChild 0 >> >> >> >> </IfModule> >> >> >> >> ``` >> >> >> >> However it stills throws more errors (~8k) than 2.4.6 on the first >> >> >> >> set >> >> >> >> of settings. This problem occurs only when using keepalive. There >> >> >> >> are >> >> >> >> no errors when using ab without -k option, although speed is lower. >> >> >> >> I >> >> >> >> can replicate this issue on newest httpd build from source (2.4.57). >> >> >> >> What is causing this difference of behavior? How can I achieve >> >> >> >> performance from 2.4.6 on 2.4.37 / 2.4.57 without throwing much more >> >> >> >> resources at httpd? These errors seems to be a root cause of a >> >> >> >> problem >> >> >> >> we have with 502 errors thrown from downstream reverse proxy server >> >> >> >> when httpd kills keepalive connection prematurely and the proxy is >> >> >> >> trying to use this connection. >> >> >> >> >> >> >> >> Below results of ab tests: >> >> >> >> >> >> >> >> Centos 7 VM: >> >> >> >> ``` >> >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://centos/ >> >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, >> >> >> >> http://www.zeustech.net/ >> >> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> >> >> >> >> >> >> Benchmarking 10.1.3.3 (be patient) >> >> >> >> Completed 100000 requests >> >> >> >> Completed 200000 requests >> >> >> >> Completed 300000 requests >> >> >> >> Completed 400000 requests >> >> >> >> Completed 500000 requests >> >> >> >> Completed 600000 requests >> >> >> >> Completed 700000 requests >> >> >> >> Completed 800000 requests >> >> >> >> Completed 900000 requests >> >> >> >> Completed 1000000 requests >> >> >> >> Finished 1000000 requests >> >> >> >> >> >> >> >> >> >> >> >> Server Software: Apache/2.4.6 >> >> >> >> Server Hostname: 10.1.3.3 >> >> >> >> Server Port: 80 >> >> >> >> >> >> >> >> Document Path: / >> >> >> >> Document Length: 7620 bytes >> >> >> >> >> >> >> >> Concurrency Level: 1000 >> >> >> >> Time taken for tests: 15.285 seconds >> >> >> >> Complete requests: 1000000 >> >> >> >> Failed requests: 67 >> >> >> >> (Connect: 0, Receive: 0, Length: 67, Exceptions: 0) >> >> >> >> Write errors: 0 >> >> >> >> Keep-Alive requests: 990567 >> >> >> >> Total transferred: 7919057974 bytes >> >> >> >> HTML transferred: 7619489460 bytes >> >> >> >> Requests per second: 65422.95 [#/sec] (mean) >> >> >> >> Time per request: 15.285 [ms] (mean) >> >> >> >> Time per request: 0.015 [ms] (mean, across all concurrent >> >> >> >> requests) >> >> >> >> Transfer rate: 505945.41 [Kbytes/sec] received >> >> >> >> >> >> >> >> Connection Times (ms) >> >> >> >> min mean[+/-sd] median max >> >> >> >> Connect: 0 0 4.7 0 1042 >> >> >> >> Processing: 3 15 16.8 13 467 >> >> >> >> Waiting: 0 14 16.2 13 433 >> >> >> >> Total: 3 15 17.7 13 1081 >> >> >> >> >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> >> >> 50% 13 >> >> >> >> 66% 15 >> >> >> >> 75% 16 >> >> >> >> 80% 17 >> >> >> >> 90% 21 >> >> >> >> 95% 23 >> >> >> >> 98% 32 >> >> >> >> 99% 44 >> >> >> >> 100% 1081 (longest request) >> >> >> >> ``` >> >> >> >> >> >> >> >> Rocky 8 VM: >> >> >> >> ``` >> >> >> >> ab -k -t 900 -c 1000 -n 1000000 http://rocky/ >> >> >> >> This is ApacheBench, Version 2.3 <$Revision: 1430300 $> >> >> >> >> Copyright 1996 Adam Twiss, Zeus Technology Ltd, >> >> >> >> http://www.zeustech.net/ >> >> >> >> Licensed to The Apache Software Foundation, http://www.apache.org/ >> >> >> >> >> >> >> >> Benchmarking 10.1.3.11 (be patient) >> >> >> >> Completed 100000 requests >> >> >> >> Completed 200000 requests >> >> >> >> Completed 300000 requests >> >> >> >> Completed 400000 requests >> >> >> >> Completed 500000 requests >> >> >> >> Completed 600000 requests >> >> >> >> Completed 700000 requests >> >> >> >> Completed 800000 requests >> >> >> >> Completed 900000 requests >> >> >> >> Completed 1000000 requests >> >> >> >> Finished 1000000 requests >> >> >> >> >> >> >> >> >> >> >> >> Server Software: Apache/2.4.37 >> >> >> >> Server Hostname: 10.1.3.11 >> >> >> >> Server Port: 80 >> >> >> >> >> >> >> >> Document Path: / >> >> >> >> Document Length: 7620 bytes >> >> >> >> >> >> >> >> Concurrency Level: 1000 >> >> >> >> Time taken for tests: 19.101 seconds >> >> >> >> Complete requests: 1000000 >> >> >> >> Failed requests: 93159 >> >> >> >> (Connect: 0, Receive: 0, Length: 85029, Exceptions: 8130) >> >> >> >> Write errors: 0 >> >> >> >> Keep-Alive requests: 912753 >> >> >> >> Total transferred: 7248228337 bytes >> >> >> >> HTML transferred: 6973694460 bytes >> >> >> >> Requests per second: 52352.12 [#/sec] (mean) >> >> >> >> Time per request: 19.101 [ms] (mean) >> >> >> >> Time per request: 0.019 [ms] (mean, across all concurrent >> >> >> >> requests) >> >> >> >> Transfer rate: 370566.51 [Kbytes/sec] received >> >> >> >> >> >> >> >> Connection Times (ms) >> >> >> >> min mean[+/-sd] median max >> >> >> >> Connect: 0 4 54.4 0 3022 >> >> >> >> Processing: 0 15 20.6 13 1400 >> >> >> >> Waiting: 0 13 17.7 12 1400 >> >> >> >> Total: 0 19 59.0 13 3048 >> >> >> >> >> >> >> >> Percentage of the requests served within a certain time (ms) >> >> >> >> 50% 13 >> >> >> >> 66% 15 >> >> >> >> 75% 17 >> >> >> >> 80% 19 >> >> >> >> 90% 24 >> >> >> >> 95% 41 >> >> >> >> 98% 53 >> >> >> >> 99% 106 >> >> >> >> 100% 3048 (longest request) >> >> >> >> ``` >> >> >> >> >> >> >> >> --- >> >> >> >> Mateusz Kempski >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> >> >> >> For additional commands, e-mail: users-h...@httpd.apache.org >> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> >> >> For additional commands, e-mail: users-h...@httpd.apache.org >> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> >> For additional commands, e-mail: users-h...@httpd.apache.org >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org >> For additional commands, e-mail: users-h...@httpd.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
