On Mon, Oct 11, 2021 at 2:32 PM <[email protected]> wrote:
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=65626
>
> Bug ID: 65626
> Summary: MPM Event doesn't shutdown idle children after working
> under high load
> Product: Apache httpd-2
> Version: 2.4.37
> Hardware: PC
> Status: NEW
> Severity: normal
> Priority: P2
> Component: mpm_event
> Assignee: [email protected]
> Reporter: [email protected]
> Target Milestone: ---
>
> After running ab with concurrency equal or more than MaxRequestWorkers for
> some
> time, apache doesn't kill idle processes any longer. Following is logged every
> second:
>
> [Mon Oct 11 15:17:29.416964 2021] [mpm_event:trace5] [pid 71:tid
> 140381265582400] event.c(2834): Not shutting down child: total daemons 16 /
> active limit 16 / ServerLimit 16
At https://github.com/apache/httpd/blob/trunk/server/mpm/event/event.c#L3181
we handle MaxSpareThreads like this:
if (idle_thread_count > max_spare_threads / num_buckets)
{
/*
* Child processes that we ask to shut down won't die immediately
* but may stay around for a long time when they finish their
* requests. If the server load changes many times, many such
* gracefully finishing processes may accumulate, filling up the
* scoreboard. To avoid running out of scoreboard entries, we
* don't shut down more processes when the total number of processes
* is high.
* [...]
*/
if (retained->total_daemons <= active_daemons_limit &&
retained->total_daemons < server_limit) {
/* Kill off one child */
ap_mpm_podx_signal(retained->buckets[child_bucket].pod,
AP_MPM_PODX_GRACEFUL);
retained->idle_spawn_rate[child_bucket] = 1;
} else {
ap_log_error(APLOG_MARK, APLOG_TRACE5, 0, ap_server_conf,
"Not shutting down child: total daemons %d / "
[...]);
}
}
with active_daemons_limit = MaxRequestWorkers / ThreadsPerChild (thus
a constant), and retained->total_daemons which is
incremented/decremented when a child forks/exits.
So it seems that besides MaxRequestsPerChild, there is nothing to
recover after active_daemons_limit was reached at one point in time.
Shouldn't we also compute a busy_thread_count and kill children
if/when later things settle down and thus busy_thread_count <
max_spare_threads (like in the attached patch)?
Regards;
Yann.
Index: server/mpm/event/event.c
===================================================================
--- server/mpm/event/event.c (revision 1894074)
+++ server/mpm/event/event.c (working copy)
@@ -3094,6 +3094,7 @@ static void perform_idle_server_maintenance(int ch
{
int i, j;
int idle_thread_count = 0;
+ int busy_thread_count = 0;
worker_score *ws;
process_score *ps;
int free_length = 0;
@@ -3139,6 +3140,9 @@ static void perform_idle_server_maintenance(int ch
++idle_thread_count;
}
if (status >= SERVER_READY && status < SERVER_GRACEFUL) {
+ if (status != SERVER_READY) {
+ ++busy_thread_count;
+ }
++child_threads_active;
}
}
@@ -3187,7 +3191,7 @@ static void perform_idle_server_maintenance(int ch
* gracefully finishing processes may accumulate, filling up the
* scoreboard. To avoid running out of scoreboard entries, we
* don't shut down more processes when the total number of processes
- * is high.
+ * is high, until there are less than MaxSpareThreads busy threads.
*
* XXX It would be nice if we could
* XXX - kill processes without keepalive connections first
@@ -3195,8 +3199,9 @@ static void perform_idle_server_maintenance(int ch
* XXX depending on server load, later be able to resurrect them
* or kill them
*/
- if (retained->total_daemons <= active_daemons_limit &&
- retained->total_daemons < server_limit) {
+ if (busy_thread_count < max_spare_threads / num_buckets
+ || (retained->total_daemons <= active_daemons_limit
+ && retained->total_daemons < server_limit)) {
/* Kill off one child */
ap_mpm_podx_signal(retained->buckets[child_bucket].pod,
AP_MPM_PODX_GRACEFUL);
@@ -3204,9 +3209,9 @@ static void perform_idle_server_maintenance(int ch
} else {
ap_log_error(APLOG_MARK, APLOG_TRACE5, 0, ap_server_conf,
"Not shutting down child: total daemons %d / "
- "active limit %d / ServerLimit %d",
+ "active limit %d / ServerLimit %d / busy threads %d",
retained->total_daemons, active_daemons_limit,
- server_limit);
+ server_limit, busy_thread_count);
}
}
else if (idle_thread_count < min_spare_threads / num_buckets) {
