[users@httpd] Regarding CVE-2021-40438
Hi All, I understand that, CVE-2021-40438 is fixed in httpd release 2.4.50 onwards. But I would like to know more about, how this issue can be exploitable in prior versions and can I know the commit id/patch details for this issue. Tried looking into commit details in github apache repo, but couldnt find anything specific to CVE-2021-40438. Please help me in this regard, With Regards Venkatesh
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Patrick, On Mon, Oct 18, 2021 at 10:13 PM Patrick Verdon wrote: > > Just a quick follow up - we've tried removing mod_http2 but still managed to > provoke a crash. See the error_log below when stopping/restarting after httpd > becomes unresponsive. It seems to have eliminated the "reslist_cleanup: Assertion `rl->ntotal == 0' failed" and "Aborted (6)" errors, which was the primary goal. Hopefully the other "corrupted size vs. prev_size" and "Segmentation fault (11)" errors were related but it does not seem to be the case.. > We need to be a bit more careful removing other modules to make sure they're > not used, which is more time consuming - do you think this is still worth > doing to address the issue? I can't tell this from the few pieces of information available so far. > > If you have any other suggestions let me know. Since httpd is now crashing with "Segmentation fault" (only), there is a way to get a coredump file generated for further analysis, you need to add this to your main/base httpd configuration: CoreDumpDirectory /tmp After each crash there should be a "/tmp/core" (or "/tmp/core.[pid]") file which can be analysed with the gdb debugger, by using these commands: $ gdb /usr/sbin/httpd /tmp/core[.pid] [and once in gdb with the "(gdb)" prompt] (gdb) thread apply all bt Please paste the result here. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Yann, Just a quick follow up - we've tried removing mod_http2 but still managed to provoke a crash. See the error_log below when stopping/restarting after httpd becomes unresponsive. We need to be a bit more careful removing other modules to make sure they're not used, which is more time consuming - do you think this is still worth doing to address the issue? If you have any other suggestions let me know. Thanks. Patrick -- # cat /var/log/httpd/error_log *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x55a67cc31e7f *** *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x55a67cc31e7f *** *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x55a67cc31e7f *** *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x55a67cc31e7f *** [Mon Oct 18 20:59:48.426225 2021] [core:notice] [pid 31207] AH00052: child pid 32036 exit signal Segmentation fault (11) [Mon Oct 18 20:59:48.426389 2021] [core:notice] [pid 31207] AH00052: child pid 31246 exit signal Segmentation fault (11) [Mon Oct 18 20:59:48.492282 2021] [core:notice] [pid 31207] AH00052: child pid 31253 exit signal Segmentation fault (11) [Mon Oct 18 20:59:48.492312 2021] [core:notice] [pid 31207] AH00052: child pid 32289 exit signal Segmentation fault (11) [Mon Oct 18 20:59:48.492455 2021] [mpm_prefork:notice] [pid 31207] AH00169: caught SIGTERM, shutting down [Mon Oct 18 20:59:48.631928 2021] [suexec:notice] [pid 32620] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Mon Oct 18 20:59:48.662384 2021] [lbmethod_heartbeat:notice] [pid 32626] AH02282: No slotmem from mod_heartmonitor [Mon Oct 18 20:59:48.724408 2021] [mpm_prefork:notice] [pid 32626] AH00163: Apache/2.4.51 (Amazon) OpenSSL/1.0.2k-fips configured -- resuming normal operations [Mon Oct 18 20:59:48.724430 2021] [core:notice] [pid 32626] AH00094: Command line: '/usr/sbin/httpd' [Mon Oct 18 20:59:49.724509 2021] [mpm_prefork:error] [pid 32626] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting *--* *Patrick Verdon | Founder* Web: www.youreko.com Mobile: +44 (0)7809 296438 Skype: patrick_verdon This entire communication is sent on behalf of Youreko Ltd and is strictly confidential to and for the sole use of the intended addressee. Registered in England - 7448349 On Mon, 18 Oct 2021 at 15:05, Patrick Verdon wrote: > Hi Yann, > > Many thanks for the super quick response. We'll try to remove mod_http2 > and other modules as you suggest to see if that helps. I'll get back to you > once we've had a chance to test it. > > Thanks. > > Patrick > > *--* > > *Patrick Verdon | Founder* > Web: www.youreko.com > Mobile: +44 (0)7809 296438 > Skype: patrick_verdon > > This entire communication is sent on behalf of > Youreko Ltd and is strictly confidential to and > for the sole use of the intended addressee. > > Registered in England - 7448349 > > > > On Mon, 18 Oct 2021 at 12:57, Yann Ylavic wrote: > >> Hi Patrick, >> >> On Mon, Oct 18, 2021 at 11:27 AM Patrick Verdon >> wrote: >> > >> > # cat /var/log/httpd/error_log >> > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal >> == 0' failed. >> [] >> > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: >> 0x557f94567e4f *** >> [] >> > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal >> == 0' failed. >> > [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: >> child pid 3166 exit signal Aborted (6) >> [] >> > [Sun Oct 17 15:53:47.990781 2021] [core:notice] [pid 2620] AH00052: >> child pid 2741 exit signal Segmentation fault (11) >> > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: >> 0x557f94567e4f *** >> [] >> > [Sun Oct 17 15:53:48.056599 2021] [core:notice] [pid 2620] AH00052: >> child pid 2727 exit signal Aborted (6) >> > [Sun Oct 17 15:53:48.056667 2021] [mpm_prefork:notice] [pid 2620] >> AH00169: caught SIGTERM, shutting down >> >> The log seems to show a stop then start sequence (which is possibly >> what "service httpd restart" does), anyway the stop crashes children >> processes that at some point have reserved/handled mod_proxy >> connections. >> >> We will discuss whether/how to fix this on the dev@ mailing list, in >> the meantime I'd suggest that: >> >> > [Sun Oct 17 15:53:48.180621 2021] [http2:warn] [pid 3581] AH10034: The >> mpm module (prefork.c) is not supported by mod_http2. The mpm determines >> how things are processed in your server. HTTP/2 has more demands in this >> regard and the currently selected mpm will just not do. This is an advisory >> warning. Your server will continue to work, but the HTTP/2 protocol will be >> inactive. >> >> .. you do not "LoadModule http2_module mod_http2.so" in your MPM >> prefork configuration, because due to its multithreaded nature (unlike >> MPM prefork) mod_http2 implies that mod_proxy will have to >> allocate/handle multiple simultaneous connection to the backend which >> is what
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Yann, Many thanks for the super quick response. We'll try to remove mod_http2 and other modules as you suggest to see if that helps. I'll get back to you once we've had a chance to test it. Thanks. Patrick *--* *Patrick Verdon | Founder* Web: www.youreko.com Mobile: +44 (0)7809 296438 Skype: patrick_verdon This entire communication is sent on behalf of Youreko Ltd and is strictly confidential to and for the sole use of the intended addressee. Registered in England - 7448349 On Mon, 18 Oct 2021 at 12:57, Yann Ylavic wrote: > Hi Patrick, > > On Mon, Oct 18, 2021 at 11:27 AM Patrick Verdon > wrote: > > > > # cat /var/log/httpd/error_log > > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > [] > > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** > [] > > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > > [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: > child pid 3166 exit signal Aborted (6) > [] > > [Sun Oct 17 15:53:47.990781 2021] [core:notice] [pid 2620] AH00052: > child pid 2741 exit signal Segmentation fault (11) > > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** > [] > > [Sun Oct 17 15:53:48.056599 2021] [core:notice] [pid 2620] AH00052: > child pid 2727 exit signal Aborted (6) > > [Sun Oct 17 15:53:48.056667 2021] [mpm_prefork:notice] [pid 2620] > AH00169: caught SIGTERM, shutting down > > The log seems to show a stop then start sequence (which is possibly > what "service httpd restart" does), anyway the stop crashes children > processes that at some point have reserved/handled mod_proxy > connections. > > We will discuss whether/how to fix this on the dev@ mailing list, in > the meantime I'd suggest that: > > > [Sun Oct 17 15:53:48.180621 2021] [http2:warn] [pid 3581] AH10034: The > mpm module (prefork.c) is not supported by mod_http2. The mpm determines > how things are processed in your server. HTTP/2 has more demands in this > regard and the currently selected mpm will just not do. This is an advisory > warning. Your server will continue to work, but the HTTP/2 protocol will be > inactive. > > .. you do not "LoadModule http2_module mod_http2.so" in your MPM > prefork configuration, because due to its multithreaded nature (unlike > MPM prefork) mod_http2 implies that mod_proxy will have to > allocate/handle multiple simultaneous connection to the backend which > is what is causing the crash here. > > > [Sun Oct 17 15:53:48.181146 2021] [lbmethod_heartbeat:notice] [pid 3581] > AH02282: No slotmem from mod_heartmonitor > > Likewise you probably don't need lbmethod_heartbeat and several > modules in your list, so I'd suggest that you cleanup your LoadModules > a bit, ideally to the strict minimum needed. > > > Regards; > Yann. > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > >
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Patrick, On Mon, Oct 18, 2021 at 11:27 AM Patrick Verdon wrote: > > # cat /var/log/httpd/error_log > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. [] > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** [] > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: child pid > 3166 exit signal Aborted (6) [] > [Sun Oct 17 15:53:47.990781 2021] [core:notice] [pid 2620] AH00052: child pid > 2741 exit signal Segmentation fault (11) > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** [] > [Sun Oct 17 15:53:48.056599 2021] [core:notice] [pid 2620] AH00052: child pid > 2727 exit signal Aborted (6) > [Sun Oct 17 15:53:48.056667 2021] [mpm_prefork:notice] [pid 2620] AH00169: > caught SIGTERM, shutting down The log seems to show a stop then start sequence (which is possibly what "service httpd restart" does), anyway the stop crashes children processes that at some point have reserved/handled mod_proxy connections. We will discuss whether/how to fix this on the dev@ mailing list, in the meantime I'd suggest that: > [Sun Oct 17 15:53:48.180621 2021] [http2:warn] [pid 3581] AH10034: The mpm > module (prefork.c) is not supported by mod_http2. The mpm determines how > things are processed in your server. HTTP/2 has more demands in this regard > and the currently selected mpm will just not do. This is an advisory warning. > Your server will continue to work, but the HTTP/2 protocol will be inactive. .. you do not "LoadModule http2_module mod_http2.so" in your MPM prefork configuration, because due to its multithreaded nature (unlike MPM prefork) mod_http2 implies that mod_proxy will have to allocate/handle multiple simultaneous connection to the backend which is what is causing the crash here. > [Sun Oct 17 15:53:48.181146 2021] [lbmethod_heartbeat:notice] [pid 3581] > AH02282: No slotmem from mod_heartmonitor Likewise you probably don't need lbmethod_heartbeat and several modules in your list, so I'd suggest that you cleanup your LoadModules a bit, ideally to the strict minimum needed. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
[users@httpd] Issue with Apache 2.4.51 hanging
Hi All, I'd appreciate some feedback on an issue I'm experiencing. I've spent quite some time researching the problem as it causes a serious outage in our application. I've searched the Web, Stack Overflow, this list's mail archives, the latest Apache bugs, and more, but have not been able to find any reports of a similar issue. Background. I'm running the latest Apache 2.4.51 on Amazon Linux with mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our application take about 45 seconds to complete so there is a warm-up cache procedure at regular intervals during the day which primes the varnish cache. The following steps reliably cause Apache to hang, requiring a manual restart: 1. Varnish cache is cleared, causing spike in load on httpd 2. Warm-up cache process kicks off with 2 long running requests (45 seconds each). This is a PHP application running under mod_php - each process grows up to 700 MB, so the application kills the httpd child process at the end to release the memory, using posix_kill(PID, 28). 3. Apache hangs and does not recover. Varnish serves 503s. 4. Manual restart required: service httpd restart 5. Errors in the log show that 2 children had segmentation faults, presumably the 2 with long running processes. Albeit ugly, this process has been running for a year and a half without any issues. We traced the date that crashes started to the date Apache was upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an issue in 2.4.51. See the error_log below and details about the installation. Any feedback on where to report this issue would be much appreciated. Thanks. Patrick -- # cat /var/log/httpd/error_log httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x557f94567e4f *** httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: child pid 3166 exit signal Aborted (6) [Sun Oct 17 15:53:47.990531 2021] [core:notice] [pid 2620] AH00052: child pid 3483 exit signal Aborted (6) [Sun Oct 17 15:53:47.990545 2021] [core:notice] [pid 2620] AH00052: child pid 2657 exit signal Aborted (6) [Sun Oct 17 15:53:47.990557 2021] [core:notice] [pid 2620] AH00052: child pid 2660 exit signal Aborted (6) [Sun Oct 17 15:53:47.990568 2021] [core:notice] [pid 2620] AH00052: child pid 2661 exit signal Aborted (6) [Sun Oct 17 15:53:47.990579 2021] [core:notice] [pid 2620] AH00052: child pid 3172 exit signal Aborted (6) [Sun Oct 17 15:53:47.990592 2021] [core:notice] [pid 2620] AH00052: child pid 2681 exit signal Aborted (6) [Sun Oct 17 15:53:47.990603 2021] [core:notice] [pid 2620] AH00052: child pid 3254 exit signal Aborted (6) [Sun Oct 17 15:53:47.990615 2021] [core:notice] [pid 2620] AH00052: child pid 2685 exit signal Aborted (6) [Sun Oct 17 15:53:47.990627 2021] [core:notice] [pid 2620] AH00052: child pid 2688 exit signal Aborted (6) [Sun Oct 17 15:53:47.990639 2021] [core:notice] [pid 2620] AH00052: child pid 3015 exit signal Aborted (6) [Sun Oct 17 15:53:47.990652 2021] [core:notice] [pid 2620] AH00052: child pid 2696 ex
