https://bz.apache.org/bugzilla/show_bug.cgi?id=66603
Bug ID: 66603
Summary: Apache dosn't terminate request properlly, local DOS
Product: Apache httpd-2
Version: 2.5-HEAD
Hardware: PC
OS: Linux
Status: NEW
Severity: critical
Priority: P2
Component: Core
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
I have noticed a problem with Apache not terminating connections properly when
using mod_proxy_fcgi with php-fpm. This issue leads to reaching the max_worker
limit.
Problem occurs when a bugged PHP app or intentional exploit sends requests to
itself in a loop, causing the max-children limit of php-fpm to be reached.
After some time, Apache stops terminating connections, and the Apache process
hangs on the connection to php-fpm due to the max-children limit being reached.
I accidentally discovered this issue when investigating why Apache reached the
max_workers limit. I observed numerous close_wait connections that kept
growing. By using strace, I determined that these close_wait connections were
hanging on the connection to php-fpm, as shown in the example trace:
[pid 2073975] 20:22:03 connect(2630, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073974] 20:22:03 connect(2573, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073973] 20:22:03 connect(2629, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073972] 20:22:03 connect(2594, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073971] 20:22:03 connect(2615, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073970] 20:22:03 connect(2633, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073969] 20:22:03 connect(2587, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073968] 20:22:03 connect(2570, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073967] 20:22:03 connect(2603, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073966] 20:22:03 connect(2623, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073965] 20:22:03 connect(2572, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073964] 20:22:03 connect(2609, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073963] 20:22:03 connect(2586, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073962] 20:22:03 connect(2616, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073961] 20:22:03 connect(2607, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073960] 20:22:03 connect(2617, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073959] 20:22:03 connect(2621, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073958] 20:22:03 connect(2605, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073957] 20:22:03 connect(2611, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073956] 20:22:03 connect(2627, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073955] 20:22:03 connect(2625, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073954] 20:22:03 connect(2619, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073953] 20:22:03 connect(2581, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073952] 20:22:03 connect(2602, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073951] 20:22:03 connect(2579, {sa_family=AF_UNIX,
sun_path="/usr/local/php73-fpm/sockets/serwer170663.sock"}, 48 <unfinished ...>
[pid 2073949] 20:22:03 futex(0x7fb517e1ee84, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073948] 20:22:03 futex(0x7fb517e1ed88, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073947] 20:22:03 futex(0x7fb517e1ec94, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073946] 20:22:03 futex(0x7fb517e1eb98, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073945] 20:22:03 futex(0x7fb517e1eaa4, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073944] 20:22:03 futex(0x7fb517e1e9a8, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073943] 20:22:03 futex(0x7fb517e1e8b4, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073942] 20:22:03 futex(0x7fb517e1e7b8, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073941] 20:22:03 futex(0x7fb517e1e6c0, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073940] 20:22:03 futex(0x7fb517e1e5cc, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073939] 20:22:03 futex(0x7fb517e1e4d0, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073938] 20:22:03 futex(0x7fb517e1e3d8, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073937] 20:22:03 futex(0x7fb517e1e2e4, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073936] 20:22:03 futex(0x7fb517e1e1e8, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073935] 20:22:03 futex(0x7fb517e1e0f0, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073934] 20:22:03 futex(0x7fb517e1dff8, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073933] 20:22:03 futex(0x7fb517e1df00, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073932] 20:22:03 futex(0x7fb517e1de0c, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073931] 20:22:03 futex(0x7fb517e1dd10, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073930] 20:22:03 futex(0x7fb517e1dc18, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073929] 20:22:03 futex(0x7fb517e1db24, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073928] 20:22:03 futex(0x7fb517e1da2c, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073927] 20:22:03 futex(0x7fb517e1d930, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073926] 20:22:03 futex(0x7fb517e1d838, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073925] 20:22:03 futex(0x7fb517e1d744, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073924] 20:22:03 futex(0x7fb517e1d648, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073923] 20:22:03 futex(0x7fb517e1d550, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073922] 20:22:03 futex(0x7fb517e1d458, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073921] 20:22:03 futex(0x7fb517e1d360, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073920] 20:22:03 futex(0x7fb518166fe0, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073919] 20:22:03 futex(0x7fb518166ee8, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
[pid 2073918] 20:22:03 futex(0x7fb518166df0, FUTEX_WAIT_PRIVATE, 0, NULL
<unfinished ...>
Unfortunately, php-fpm is not able terminate these connections.
In my Apache configuration, I have set Timeout and ProxyTimeout to 300 seconds,
so according to the configuration, Apache should terminate the connection after
this time. However, it does not happen even when Apache responds correctly, and
the server is not overloaded.
To reproduce the issue, I have written a code snippet that simulates the
problem:
<?php
$url = 'test2.lh.pl';
$port = 80;
while(1) {
$socket = stream_socket_client("tcp://$url:$port", $errno, $errstr, 30);
stream_set_timeout($socket, 10);
$request = "GET / HTTP/1.1\r\n";
$request .= "Host: $url\r\n";
$request .= "Connection: close\r\n\r\n";
fwrite($socket, $request);
$response = '';
while (!feof($socket)) {
$response .= fread($socket, 8192);
$info = stream_get_meta_data($socket);
if ($info['timed_out']) {
stream_socket_shutdown($socket, STREAM_SHUT_WR);
break;
}
}
echo $response;
fclose($socket);
}
?>
To reproduce the problem, open a site with this file, wait for a 504 error, and
then refresh the site. After a while, you will notice that in netstat -tnp,
connections stop terminating and only keep growing. Wait for 300 seconds (or
the timeout value you have set) when Apache should terminate the connection,
but it doesn't, even though Apache and the server are fully responsive. You can
visit another site or do something else during this time. This issue is
especially troublesome on shared servers because the max_workers limit will be
reached, causing other sites to stop working.
Here are my Apache and PHP configurations:
Vhost.conf
<VirtualHost *:80>
ServerName test2.lh.pl
ServerAlias www.test2.lh.pl
DocumentRoot /home/test2
<Directory /home/test2>
AllowOverride All
Require all granted
</Directory>
<FilesMatch ".php$">
SetHandler "proxy:unix:/run/php/test2.sock|fcgi://localhost"
</FilesMatch>
</VirtualHost>
apache2.conf
Evrything defult like in debian 11 expect:
Timeout 300
ProxyTimeout 300
mpm-event.conf
<IfModule mpm_event_module>
StartServers 2
ServerLimit 64
MinSpareThreads 75
MaxSpareThreads 250
ThreadLimit 64
ThreadsPerChild 32
MaxRequestWorkers 2048
MaxConnectionsPerChild 0
</IfModule>
php-fpm pool socket conf:
[test2]
user = $pool
group = $pool
listen = /run/php/$pool.sock
listen.owner = $pool
listen.group = www-data
listen.mode = 660
env[TMP] = /tmp
env[TMPDIR] = /tmp
env[TEMP] = /tmp
php_admin_value[sys_temp_dir] = /tmp
php_admin_value[upload_tmp_dir] = /tmp
php_admin_value[session.save_path] = /tmp
pm = ondemand
pm.max_children = 20
pm.max_requests = 500
pm.process_idle_timeout = 20
pm.status_path = /status
request_terminate_timeout = 300
security.limit_extensions = .php .php52 .php53 .php54 .php55 .php56 .php60
.php70 .php71
access.log = /var/log/php-fpm/access.log
access.format = %{REMOTE_ADDR}e %n %T %m %r%Q%q %s %f %{REQUEST_URI}e %{mili}d
%{kilo}M %C 74 %{REQUEST_SCHEME}e://%{HTTP_HOST}e%{REQUEST_URI}e
%{GEOIP_CONTINENT_CODE}e %{GEOIP_COUNTRY_CODE}e
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
