https://bz.apache.org/bugzilla/show_bug.cgi?id=69901
Bug ID: 69901
Summary: mod_file_cache not thread safe
Product: Apache httpd-2
Version: 2.5-HEAD
Hardware: PC
OS: Linux
Status: NEW
Severity: major
Priority: P2
Component: mod_file_cache
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
Summary: mod_file_cache MMapFile directive causes empty body replies after
being under load when in a multi-thread, multi-core configuration.
Version/Platform: httpd versions trunk,2.4.58 both locally built, and apt
distributed httpd 2.4.58, APR version 1.7.2, Ubuntu 24.04.3 LTS.
Running using this configuration file:
ServerRoot "/opt/httpd-test"
Listen 80
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule file_cache_module modules/mod_file_cache.so
LoadModule unixd_module modules/mod_unixd.so
<IfModule unixd_module>
User ubuntu
Group ubuntu
</IfModule>
ServerAdmin [email protected]
Define DOCROOT "/home/ubuntu/dummy-site/"
DocumentRoot "${DOCROOT}"
<Directory "${DOCROOT}">
Options Indexes FollowSymLinks
AllowOverride None
</Directory>
ErrorLog "logs/error_log"
LogLevel warn
KeepAlive Off
MMapFile /home/ubuntu/dummy-site/small/page-00001.html
(page-00001.html is just a normal html file of ~200KiB)
Reproduction:
1. Run httpd using at least 2 cores, using any mpm that is not prefork, and
with some file mapped via the MMapFile directive.
2. Run any fast benchmarking utility to fetch the mapped file, I used "siege -c
8 -b -i http://localhost/small/page-00001.html";
Actual Results:
siege almost instantly(<0.1s) bails due to too many failed transactions.
Any future request, for example via curl, returns an empty reply.
Expected Behavior:
Siege should not observe any failed transactions and curl should keep returning
the file body.
Observations and workaround:
Limiting httpd to one core via "taskset -c 2 sudo /opt/httpd-test/bin/apachectl
-X" successfully prevents this problem from appearing, suggesting the issue is
thread safety.
Tested with both prefork,worker and event MPMs and without limiting to one
core, only prefork does not have this behaviour.
When requests start failing, smaps and strace both show that the file is no
longer mapped.
Root Cause:
I have already investigated the issue and identified that it is in
mod_file_cache.c:278-279:
apr_mmap_dup(&mm, file->mm, r->pool);
b = apr_bucket_mmap_create(mm, 0, (apr_size_t)file->finfo.size,
c->bucket_alloc);
It appears that concurrent modification of the apr_mmap_t ring during creation
and destruction of the mmap buckets causes the mapped file to be unmapped
unnecessarily.
This issue has likely always existed in this module.
Fix:
I fixed this in my branch by replacing the 2 lines with:
b = apr_bucket_immortal_create((const char *)file->mm->mm,
(apr_size_t)file->finfo.size, c->bucket_alloc);
The file remains mapped for the entirety of the server runtime, so using an
immortal bucket for it is okay and solves this issue.
I will be making a pull request on github with the fix after some stress
testing.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
