Re: [users@httpd] Issue with Apache 2.4.51 hanging
Thanks @William for the suggestion. I haven't tried strace, I'll look into it, although I've now experienced the crash without the 2 long running processes - so it seems like it may be just the application under load when the varnish cache is first cleared that causes the problem, which is not very encouraging.. @Yann thanks for the pointer, I'll get in touch with the PHP maintainers to see if they've had any similar reports. Thanks again for your time and feedback - much appreciated. Patrick *--* *Patrick Verdon | Founder* Web: www.youreko.com Mobile: +44 (0)7809 296438 Skype: patrick_verdon This entire communication is sent on behalf of Youreko Ltd and is strictly confidential to and for the sole use of the intended addressee. Registered in England - 7448349 On Wed, 27 Oct 2021 at 12:47, Yann Ylavic wrote: > On Tue, Oct 26, 2021 at 7:36 PM Patrick Verdon > wrote: > > > > Do you know who maintains mod_php, is it worth following up with them? > > I'd suggest reporting the issue to the php maintainers ( > https://bugs.php.net/). > It may ring a bell there.. > > Regards; > Yann. > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > >
Re: [users@httpd] Issue with Apache 2.4.51 hanging
On Tue, Oct 26, 2021 at 7:36 PM Patrick Verdon wrote: > > Do you know who maintains mod_php, is it worth following up with them? I'd suggest reporting the issue to the php maintainers (https://bugs.php.net/). It may ring a bell there.. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Issue with Apache 2.4.51 hanging
> Op 18 okt. 2021 om 11:27 heeft Patrick Verdon > het volgende geschreven: > > > Hi All, > > I'd appreciate some feedback on an issue I'm experiencing. I've spent quite > some time researching the problem as it causes a serious outage in our > application. I've searched the Web, Stack Overflow, this list's mail > archives, the latest Apache bugs, and more, but have not been able to find > any reports of a similar issue. > > Background. I'm running the latest Apache 2.4.51 on Amazon Linux with > mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our > application take about 45 seconds to complete so there is a warm-up cache > procedure at regular intervals during the day which primes the varnish cache. > The following steps reliably cause Apache to hang, requiring a manual restart: > Varnish cache is cleared, causing spike in load on httpd > Warm-up cache process kicks off with 2 long running requests (45 seconds > each). This is a PHP application running under mod_php - each process grows > up to 700 MB, so the application kills the httpd child process at the end to > release the memory, using posix_kill(PID, 28). > Apache hangs and does not recover. Varnish serves 503s. Step 3 seems to be a result of the failed step 2. Have you tried stracing the long-running script on CLI? > Manual restart required: service httpd restart > Errors in the log show that 2 children had segmentation faults, presumably > the 2 with long running processes. > > Albeit ugly, this process has been running for a year and a half without any > issues. We traced the date that crashes started to the date Apache was > upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an issue > in 2.4.51. > > See the error_log below and details about the installation. > > Any feedback on where to report this issue would be much appreciated. > > Thanks. > > Patrick > > -- > > # cat /var/log/httpd/error_log > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: child pid > 3166 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990531 2021] [core:notice] [pid 2620] AH00052: child pid > 3483 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990545 2021] [core:notice] [pid 2620] AH00052: child pid > 2657 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990557 2021] [core:notice] [pid 2620] AH00052: child pid > 2660 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990568 2021] [core:notice] [pid 2620] AH00052: child pid > 2661 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990579 2021] [core:notice] [pid 2620] AH00052: child pid > 3172 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990592 2021] [core:notice] [pid 2620] AH00052: child pid > 2681 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990603 2021] [core:notice] [pid 2620] AH00052: child pid > 3254 exit signal Aborted (6) >
Re: [users@httpd] Issue with Apache 2.4.51 hanging
I appreciate saying that "nothing else has changed" is not particularly helpful - although on reflection, this is actually not quite true, a number of other system updates were applied as well as httpd, listed below. The PHP installation was not updated. Unfortunately I don't have much more to go on than the fact that the day after the updates we started experiencing regular crashes, having not had any downtime for at least 12 months previously. This seemed like too much of a coincidence. The only other thing that may have "changed" is that our traffic increased in the few weeks before the first crash. If it's difficult with this configuration to differentiate between third party libraries and httpd being the problem, it sounds like it's going to be a tricky issue to diagnose. Do you know who maintains mod_php, is it worth following up with them? If there aren't any other possible courses of action then we may need to consider php-fpm as a last resort after all! Thanks. Patrick -- *For reference, in addition to httpd 2.4.48, the following packages were updated on the day before the first crash* $ yum list updates Loaded plugins: priorities, update-motd, upgrade-helper amzn-main | 2.1 kB 00:00:00 amzn-updates | 3.8 kB 00:00:00 Updated Packages cloud-init.noarch 0.7.6-43.23.amzn1 amzn-updates curl.x86_64 7.61.1-12.99.amzn1amzn-updates ec2-net-utils.noarch 0.7-43.5.amzn1amzn-updates ec2-utils.noarch 0.7-43.5.amzn1amzn-updates glibc.x86_64 2.17-324.188.amzn1amzn-updates glibc-common.x86_64 2.17-324.188.amzn1amzn-updates glibc-devel.x86_64 2.17-324.188.amzn1amzn-updates glibc-headers.x86_64 2.17-324.188.amzn1amzn-updates grub.x86_64 1:0.97-94.32.amzn1amzn-updates iptables.x86_64 1.4.21-34.33.amzn1amzn-updates kernel.x86_64 4.14.248-129.473.amzn1amzn-updates libcurl.x86_64 7.61.1-12.99.amzn1amzn-updates nspr.x86_64 4.25.0-2.45.amzn1 amzn-updates nss.x86_64 3.53.1-7.85.amzn1 amzn-updates nss-softokn.x86_64 3.53.1-6.46.amzn1 amzn-updates nss-softokn-freebl.x86_64 3.53.1-6.46.amzn1 amzn-updates nss-sysinit.x86_64 3.53.1-7.85.amzn1 amzn-updates nss-tools.x86_64 3.53.1-7.85.amzn1 amzn-updates nss-util.x86_64 3.53.1-1.58.amzn1 amzn-updates openssh.x86_64 7.4p1-21.75.amzn1 amzn-updates openssh-clients.x86_64 7.4p1-21.75.amzn1 amzn-updates openssh-server.x86_64 7.4p1-21.75.amzn1 amzn-updates openssl.x86_64 1:1.0.2k-16.154.amzn1 amzn-updates python27.x86_64 2.7.18-2.141.amzn1amzn-updates python27-devel.x86_64 2.7.18-2.141.amzn1amzn-updates python27-libs.x86_64 2.7.18-2.141.amzn1amzn-updates python27-pyliblzma.x86_64 0.5.3-11.7.amzn1 amzn-updates python27-setuptools.noarch 36.2.7-1.35.amzn1 amzn-updates rpm.x86_64 4.11.3-40.79.amzn1amzn-updates rpm-build-libs.x86_64 4.11.3-40.79.amzn1amzn-updates rpm-libs.x86_64 4.11.3-40.79.amzn1amzn-updates rpm-python27.x86_64 4.11.3-40.79.amzn1amzn-updates ruby20.x86_64 2.0.0.648-2.40.amzn1 amzn-updates ruby20-irb.noarch 2.0.0.648-2.40.amzn1 amzn-updates ruby20-libs.x86_64 2.0.0.648-2.40.amzn1 amzn-updates rubygem20-bigdecimal.x86_64 1.2.0-2.40.amzn1 amzn-updates rubygem20-psych.x86_64 2.0.0-2.40.amzn1 amzn-updates rubygems20.noarch 2.0.14.1-2.40.amzn1 amzn-updates screen.x86_64 4.0.3-19.7.amzn1 amzn-updates tzdata.noarch 2021a-1.79.amzn1 amzn-updates tzdata-java.noarch 2021a-1.79.amzn1
Re: [users@httpd] Issue with Apache 2.4.51 hanging
I think we are deviating a bit offtopic here. but still just a little note: mpm_prefork is not problematic, what is problematic is when people do not size httpd with mpm_prefork to deal with the amount of incoming requests they receive. processes are more expensive to generate than threads and on very busy servers with prefork one must "pre-spawn" lots of processes beforehand to avoid delays and make sure there are enough spare servers waiting to deal with load spikes. - The matter here is, when you load mod_php and burden httpd with third party libraries and force it to use a non-threaded mpm, it is hard to differentiate when httpd is being problematic and when the third party libraries are causing the issue. Op here mentions he has updated httpd but not php module and thus assumes httpd is at fault because "nothing else has changed" yet in the debugging what I mostly see is zend_execute_scripts () which belong to php. Cheers, El mar, 26 oct 2021 a las 16:26, Ruben Safir () escribió: > > mpm_prefork has been problematic, FWIW > > > On Tue, Oct 26, 2021 at 11:17:31AM +0100, Patrick Verdon wrote: > > Hi Daniel, > > > > Thanks for the feedback. > > > > Unfortunately that's not really an option for us, as it's a major > > architecture change, and is only something we would consider as a > > last resort. > > > > Using mod_php works well for us and our configuration/application has not > > changed in years - the crashes started very specifically when we upgraded > > to Apache 2.4.48 from 2.4.46. > > > > From what I can see, mod_php is still used widely and I assume it is > > therefore still a supported module? > > > > I'd be grateful for any other suggestions on how to get to the bottom of > > the issue. > > > > Thanks. > > > > Patrick > > > > *--* > > > > *Patrick Verdon | Founder* > > Web: www.youreko.com > > Mobile: +44 (0)7809 296438 > > Skype: patrick_verdon > > > > This entire communication is sent on behalf of > > Youreko Ltd and is strictly confidential to and > > for the sole use of the intended addressee. > > > > Registered in England - 7448349 > > > > > > > > On Tue, 26 Oct 2021 at 09:56, Daniel Ferradal wrote: > > > > > Hello, > > > > > > Seems it is related to third party php module being loaded in Apache > > > HTTPD, no? > > > > > > It would be awesome if you can move your php files to be dealt with > > > php-fpm instead and from apache just proxy to php-fpm.. and see if > > > your Apache server would ever hang again (if you do it you could even use > > > mpm_event instead of prefork to have http/2 working). > > > > > > Cheers > > > > > > El vie, 22 oct 2021 a las 18:16, Patrick Verdon (< > > > patrick.ver...@youreko.com>) escribi??: > > > > > >> Hi Yann, > > >> > > >> I finally managed to provoke the crash - there were 6 core dumps, I > > >> followed your instructions and have pasted the gdb output below for each > > >> one. > > >> > > >> Let me know if this sheds any light on the problem. > > >> > > >> Thanks. > > >> > > >> Patrick > > >> > > >> -- > > >> > > >> *# gdb /usr/sbin/httpd /tmp/core.1221* > > >> Thread 2 (Thread 0x7f5876bff840 (LWP 1221)): > > >> #0 0x7f587518e687 in kill () from /lib64/libc.so.6 > > >> #1 > > >> #2 0x7f5861def0d8 in ?? () from /lib64/libgcc_s.so.1 > > >> #3 0x7f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1 > > >> #4 0x7f587526cbf6 in backtrace () from /lib64/libc.so.6 > > >> #5 0x7f58751d0fb4 in __libc_message () from /lib64/libc.so.6 > > >> #6 0x7f58751d8854 in malloc_consolidate () from /lib64/libc.so.6 > > >> #7 0x7f58751d923e in _int_free () from /lib64/libc.so.6 > > >> #8 0x7f5875963ded in apr_allocator_destroy () from > > >> /usr/lib64/libapr-1.so.0 > > >> #9 0x7f586c276477 in ?? () from > > >> /etc/httpd/modules/mod_mpm_prefork.so > > >> #10 0x7f586c2764cb in ?? () from > > >> /etc/httpd/modules/mod_mpm_prefork.so > > >> #11 > > >> #12 0x7f587524bcfd in poll () from /lib64/libc.so.6 > > >> #13 0x7f5868086682 in ?? () from /etc/httpd/modules/libphp-7.0.so > > >> #14 0x7f5867f8fb8b in ?? () from /etc/httpd/modules/libphp-7.0.so > > >> #15 0x7f586807ab3e in _php_stream_fill_read_buffer () from > > >> /etc/httpd/modules/libphp-7.0.so > > >> #16 0x7f586807afbb in _php_stream_get_line () from > > >> /etc/httpd/modules/ > > >> libphp-7.0.so > > >> #17 0x7f586805012f in ?? () from /etc/httpd/modules/libphp-7.0.so > > >> #18 0x7f5868052148 in ?? () from /etc/httpd/modules/libphp-7.0.so > > >> #19 0x7f586807d079 in _php_stream_open_wrapper_ex () from > > >> /etc/httpd/modules/libphp-7.0.so > > >> #20 0x7f586801181b in ?? () from /etc/httpd/modules/libphp-7.0.so > > >> #21 0x7f585e957b3c in ?? () > > >> #22 0x7f5866fcde00 in ?? () > > >> #23 0x1c07 in ?? () > > >> #24 0x7f585605b380 in ?? () > > >> #25 0x7f585605b3a0 in ?? () > > >> #26 0x7f5854859bd8 in ?? () > > >> #27 0x7f5866fefe00 in ?? () > > >> #28
Re: [users@httpd] Issue with Apache 2.4.51 hanging
mpm_prefork has been problematic, FWIW On Tue, Oct 26, 2021 at 11:17:31AM +0100, Patrick Verdon wrote: > Hi Daniel, > > Thanks for the feedback. > > Unfortunately that's not really an option for us, as it's a major > architecture change, and is only something we would consider as a > last resort. > > Using mod_php works well for us and our configuration/application has not > changed in years - the crashes started very specifically when we upgraded > to Apache 2.4.48 from 2.4.46. > > From what I can see, mod_php is still used widely and I assume it is > therefore still a supported module? > > I'd be grateful for any other suggestions on how to get to the bottom of > the issue. > > Thanks. > > Patrick > > *--* > > *Patrick Verdon | Founder* > Web: www.youreko.com > Mobile: +44 (0)7809 296438 > Skype: patrick_verdon > > This entire communication is sent on behalf of > Youreko Ltd and is strictly confidential to and > for the sole use of the intended addressee. > > Registered in England - 7448349 > > > > On Tue, 26 Oct 2021 at 09:56, Daniel Ferradal wrote: > > > Hello, > > > > Seems it is related to third party php module being loaded in Apache > > HTTPD, no? > > > > It would be awesome if you can move your php files to be dealt with > > php-fpm instead and from apache just proxy to php-fpm.. and see if > > your Apache server would ever hang again (if you do it you could even use > > mpm_event instead of prefork to have http/2 working). > > > > Cheers > > > > El vie, 22 oct 2021 a las 18:16, Patrick Verdon (< > > patrick.ver...@youreko.com>) escribi??: > > > >> Hi Yann, > >> > >> I finally managed to provoke the crash - there were 6 core dumps, I > >> followed your instructions and have pasted the gdb output below for each > >> one. > >> > >> Let me know if this sheds any light on the problem. > >> > >> Thanks. > >> > >> Patrick > >> > >> -- > >> > >> *# gdb /usr/sbin/httpd /tmp/core.1221* > >> Thread 2 (Thread 0x7f5876bff840 (LWP 1221)): > >> #0 0x7f587518e687 in kill () from /lib64/libc.so.6 > >> #1 > >> #2 0x7f5861def0d8 in ?? () from /lib64/libgcc_s.so.1 > >> #3 0x7f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1 > >> #4 0x7f587526cbf6 in backtrace () from /lib64/libc.so.6 > >> #5 0x7f58751d0fb4 in __libc_message () from /lib64/libc.so.6 > >> #6 0x7f58751d8854 in malloc_consolidate () from /lib64/libc.so.6 > >> #7 0x7f58751d923e in _int_free () from /lib64/libc.so.6 > >> #8 0x7f5875963ded in apr_allocator_destroy () from > >> /usr/lib64/libapr-1.so.0 > >> #9 0x7f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > >> #10 0x7f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > >> #11 > >> #12 0x7f587524bcfd in poll () from /lib64/libc.so.6 > >> #13 0x7f5868086682 in ?? () from /etc/httpd/modules/libphp-7.0.so > >> #14 0x7f5867f8fb8b in ?? () from /etc/httpd/modules/libphp-7.0.so > >> #15 0x7f586807ab3e in _php_stream_fill_read_buffer () from > >> /etc/httpd/modules/libphp-7.0.so > >> #16 0x7f586807afbb in _php_stream_get_line () from /etc/httpd/modules/ > >> libphp-7.0.so > >> #17 0x7f586805012f in ?? () from /etc/httpd/modules/libphp-7.0.so > >> #18 0x7f5868052148 in ?? () from /etc/httpd/modules/libphp-7.0.so > >> #19 0x7f586807d079 in _php_stream_open_wrapper_ex () from > >> /etc/httpd/modules/libphp-7.0.so > >> #20 0x7f586801181b in ?? () from /etc/httpd/modules/libphp-7.0.so > >> #21 0x7f585e957b3c in ?? () > >> #22 0x7f5866fcde00 in ?? () > >> #23 0x1c07 in ?? () > >> #24 0x7f585605b380 in ?? () > >> #25 0x7f585605b3a0 in ?? () > >> #26 0x7f5854859bd8 in ?? () > >> #27 0x7f5866fefe00 in ?? () > >> #28 0x7f5853291dc0 in ?? () > >> #29 0x7f586805cf01 in ?? () from /etc/httpd/modules/libphp-7.0.so > >> #30 0x7f586805f6f4 in ?? () from /etc/httpd/modules/libphp-7.0.so > >> #31 0x7f586810c57d in ?? () from /etc/httpd/modules/libphp-7.0.so > >> #32 0x7f58680fe9eb in execute_ex () from /etc/httpd/modules/ > >> libphp-7.0.so > >> #33 0x7f586814849f in zend_execute () from /etc/httpd/modules/ > >> libphp-7.0.so > >> #34 0x7f58680c1e84 in zend_execute_scripts () from /etc/httpd/modules/ > >> libphp-7.0.so > >> #35 0x7f5868064808 in php_execute_script () from /etc/httpd/modules/ > >> libphp-7.0.so > >> #36 0x7f5868149dea in ?? () from /etc/httpd/modules/libphp-7.0.so > >> #37 0x55e5af703bc0 in ap_run_handler () > >> #38 0x55e5af704109 in ap_invoke_handler () > >> #39 0x55e5af71a3ea in ap_process_async_request () > >> #40 0x55e5af71a6be in ap_process_request () > >> #41 0x55e5af716764 in ?? () > >> #42 0x55e5af70d810 in ap_run_process_connection () > >> #43 0x7f586c276b89 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > >> #44 0x7f586c276dc0 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > >> #45 0x7f586c277bc6 in ?? () from
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Daniel, Thanks for the feedback. Unfortunately that's not really an option for us, as it's a major architecture change, and is only something we would consider as a last resort. Using mod_php works well for us and our configuration/application has not changed in years - the crashes started very specifically when we upgraded to Apache 2.4.48 from 2.4.46. >From what I can see, mod_php is still used widely and I assume it is therefore still a supported module? I'd be grateful for any other suggestions on how to get to the bottom of the issue. Thanks. Patrick *--* *Patrick Verdon | Founder* Web: www.youreko.com Mobile: +44 (0)7809 296438 Skype: patrick_verdon This entire communication is sent on behalf of Youreko Ltd and is strictly confidential to and for the sole use of the intended addressee. Registered in England - 7448349 On Tue, 26 Oct 2021 at 09:56, Daniel Ferradal wrote: > Hello, > > Seems it is related to third party php module being loaded in Apache > HTTPD, no? > > It would be awesome if you can move your php files to be dealt with > php-fpm instead and from apache just proxy to php-fpm.. and see if > your Apache server would ever hang again (if you do it you could even use > mpm_event instead of prefork to have http/2 working). > > Cheers > > El vie, 22 oct 2021 a las 18:16, Patrick Verdon (< > patrick.ver...@youreko.com>) escribió: > >> Hi Yann, >> >> I finally managed to provoke the crash - there were 6 core dumps, I >> followed your instructions and have pasted the gdb output below for each >> one. >> >> Let me know if this sheds any light on the problem. >> >> Thanks. >> >> Patrick >> >> -- >> >> *# gdb /usr/sbin/httpd /tmp/core.1221* >> Thread 2 (Thread 0x7f5876bff840 (LWP 1221)): >> #0 0x7f587518e687 in kill () from /lib64/libc.so.6 >> #1 >> #2 0x7f5861def0d8 in ?? () from /lib64/libgcc_s.so.1 >> #3 0x7f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1 >> #4 0x7f587526cbf6 in backtrace () from /lib64/libc.so.6 >> #5 0x7f58751d0fb4 in __libc_message () from /lib64/libc.so.6 >> #6 0x7f58751d8854 in malloc_consolidate () from /lib64/libc.so.6 >> #7 0x7f58751d923e in _int_free () from /lib64/libc.so.6 >> #8 0x7f5875963ded in apr_allocator_destroy () from >> /usr/lib64/libapr-1.so.0 >> #9 0x7f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so >> #10 0x7f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so >> #11 >> #12 0x7f587524bcfd in poll () from /lib64/libc.so.6 >> #13 0x7f5868086682 in ?? () from /etc/httpd/modules/libphp-7.0.so >> #14 0x7f5867f8fb8b in ?? () from /etc/httpd/modules/libphp-7.0.so >> #15 0x7f586807ab3e in _php_stream_fill_read_buffer () from >> /etc/httpd/modules/libphp-7.0.so >> #16 0x7f586807afbb in _php_stream_get_line () from /etc/httpd/modules/ >> libphp-7.0.so >> #17 0x7f586805012f in ?? () from /etc/httpd/modules/libphp-7.0.so >> #18 0x7f5868052148 in ?? () from /etc/httpd/modules/libphp-7.0.so >> #19 0x7f586807d079 in _php_stream_open_wrapper_ex () from >> /etc/httpd/modules/libphp-7.0.so >> #20 0x7f586801181b in ?? () from /etc/httpd/modules/libphp-7.0.so >> #21 0x7f585e957b3c in ?? () >> #22 0x7f5866fcde00 in ?? () >> #23 0x1c07 in ?? () >> #24 0x7f585605b380 in ?? () >> #25 0x7f585605b3a0 in ?? () >> #26 0x7f5854859bd8 in ?? () >> #27 0x7f5866fefe00 in ?? () >> #28 0x7f5853291dc0 in ?? () >> #29 0x7f586805cf01 in ?? () from /etc/httpd/modules/libphp-7.0.so >> #30 0x7f586805f6f4 in ?? () from /etc/httpd/modules/libphp-7.0.so >> #31 0x7f586810c57d in ?? () from /etc/httpd/modules/libphp-7.0.so >> #32 0x7f58680fe9eb in execute_ex () from /etc/httpd/modules/ >> libphp-7.0.so >> #33 0x7f586814849f in zend_execute () from /etc/httpd/modules/ >> libphp-7.0.so >> #34 0x7f58680c1e84 in zend_execute_scripts () from /etc/httpd/modules/ >> libphp-7.0.so >> #35 0x7f5868064808 in php_execute_script () from /etc/httpd/modules/ >> libphp-7.0.so >> #36 0x7f5868149dea in ?? () from /etc/httpd/modules/libphp-7.0.so >> #37 0x55e5af703bc0 in ap_run_handler () >> #38 0x55e5af704109 in ap_invoke_handler () >> #39 0x55e5af71a3ea in ap_process_async_request () >> #40 0x55e5af71a6be in ap_process_request () >> #41 0x55e5af716764 in ?? () >> #42 0x55e5af70d810 in ap_run_process_connection () >> #43 0x7f586c276b89 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so >> #44 0x7f586c276dc0 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so >> #45 0x7f586c277bc6 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so >> #46 0x55e5af6e3e8e in ap_run_mpm () >> #47 0x55e5af6dc0b9 in main () >> >> Thread 1 (Thread 0x7f5853d61700 (LWP 1222)): >> #0 0x7f587524da63 in select () from /lib64/libc.so.6 >> #1 0x7f587596ff65 in apr_sleep () from /usr/lib64/libapr-1.so.0 >> #2 0x7f586d1f8e62 in ?? () from /etc/httpd/modules/mod_watchdog.so >>
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hello, Seems it is related to third party php module being loaded in Apache HTTPD, no? It would be awesome if you can move your php files to be dealt with php-fpm instead and from apache just proxy to php-fpm.. and see if your Apache server would ever hang again (if you do it you could even use mpm_event instead of prefork to have http/2 working). Cheers El vie, 22 oct 2021 a las 18:16, Patrick Verdon () escribió: > Hi Yann, > > I finally managed to provoke the crash - there were 6 core dumps, I > followed your instructions and have pasted the gdb output below for each > one. > > Let me know if this sheds any light on the problem. > > Thanks. > > Patrick > > -- > > *# gdb /usr/sbin/httpd /tmp/core.1221* > Thread 2 (Thread 0x7f5876bff840 (LWP 1221)): > #0 0x7f587518e687 in kill () from /lib64/libc.so.6 > #1 > #2 0x7f5861def0d8 in ?? () from /lib64/libgcc_s.so.1 > #3 0x7f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1 > #4 0x7f587526cbf6 in backtrace () from /lib64/libc.so.6 > #5 0x7f58751d0fb4 in __libc_message () from /lib64/libc.so.6 > #6 0x7f58751d8854 in malloc_consolidate () from /lib64/libc.so.6 > #7 0x7f58751d923e in _int_free () from /lib64/libc.so.6 > #8 0x7f5875963ded in apr_allocator_destroy () from > /usr/lib64/libapr-1.so.0 > #9 0x7f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > #10 0x7f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > #11 > #12 0x7f587524bcfd in poll () from /lib64/libc.so.6 > #13 0x7f5868086682 in ?? () from /etc/httpd/modules/libphp-7.0.so > #14 0x7f5867f8fb8b in ?? () from /etc/httpd/modules/libphp-7.0.so > #15 0x7f586807ab3e in _php_stream_fill_read_buffer () from > /etc/httpd/modules/libphp-7.0.so > #16 0x7f586807afbb in _php_stream_get_line () from /etc/httpd/modules/ > libphp-7.0.so > #17 0x7f586805012f in ?? () from /etc/httpd/modules/libphp-7.0.so > #18 0x7f5868052148 in ?? () from /etc/httpd/modules/libphp-7.0.so > #19 0x7f586807d079 in _php_stream_open_wrapper_ex () from > /etc/httpd/modules/libphp-7.0.so > #20 0x7f586801181b in ?? () from /etc/httpd/modules/libphp-7.0.so > #21 0x7f585e957b3c in ?? () > #22 0x7f5866fcde00 in ?? () > #23 0x1c07 in ?? () > #24 0x7f585605b380 in ?? () > #25 0x7f585605b3a0 in ?? () > #26 0x7f5854859bd8 in ?? () > #27 0x7f5866fefe00 in ?? () > #28 0x7f5853291dc0 in ?? () > #29 0x7f586805cf01 in ?? () from /etc/httpd/modules/libphp-7.0.so > #30 0x7f586805f6f4 in ?? () from /etc/httpd/modules/libphp-7.0.so > #31 0x7f586810c57d in ?? () from /etc/httpd/modules/libphp-7.0.so > #32 0x7f58680fe9eb in execute_ex () from /etc/httpd/modules/ > libphp-7.0.so > #33 0x7f586814849f in zend_execute () from /etc/httpd/modules/ > libphp-7.0.so > #34 0x7f58680c1e84 in zend_execute_scripts () from /etc/httpd/modules/ > libphp-7.0.so > #35 0x7f5868064808 in php_execute_script () from /etc/httpd/modules/ > libphp-7.0.so > #36 0x7f5868149dea in ?? () from /etc/httpd/modules/libphp-7.0.so > #37 0x55e5af703bc0 in ap_run_handler () > #38 0x55e5af704109 in ap_invoke_handler () > #39 0x55e5af71a3ea in ap_process_async_request () > #40 0x55e5af71a6be in ap_process_request () > #41 0x55e5af716764 in ?? () > #42 0x55e5af70d810 in ap_run_process_connection () > #43 0x7f586c276b89 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > #44 0x7f586c276dc0 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > #45 0x7f586c277bc6 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > #46 0x55e5af6e3e8e in ap_run_mpm () > #47 0x55e5af6dc0b9 in main () > > Thread 1 (Thread 0x7f5853d61700 (LWP 1222)): > #0 0x7f587524da63 in select () from /lib64/libc.so.6 > #1 0x7f587596ff65 in apr_sleep () from /usr/lib64/libapr-1.so.0 > #2 0x7f586d1f8e62 in ?? () from /etc/httpd/modules/mod_watchdog.so > #3 0x7f5875731e75 in start_thread () from /lib64/libpthread.so.0 > #4 0x7f5875256a2d in clone () from /lib64/libc.so.6 > > > *# gdb /usr/sbin/httpd /tmp/core.1957* > Thread 1 (Thread 0x7f5876bff840 (LWP 1957)): > #0 0x7f5861def0d8 in ?? () from /lib64/libgcc_s.so.1 > #1 0x7f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1 > #2 0x7f587526cbf6 in backtrace () from /lib64/libc.so.6 > #3 0x7f58751d0fb4 in __libc_message () from /lib64/libc.so.6 > #4 0x7f58751d8854 in malloc_consolidate () from /lib64/libc.so.6 > #5 0x7f58751d923e in _int_free () from /lib64/libc.so.6 > #6 0x7f5875963ded in apr_allocator_destroy () from > /usr/lib64/libapr-1.so.0 > #7 0x7f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > #8 0x7f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so > #9 > #10 0x7f587524bcfd in poll () from /lib64/libc.so.6 > #11 0x7f5865f45ea9 in ?? () > #12 0x55e500d80001 in ?? () > #13 0x0006adf5 in ?? () > #14
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Yann, I finally managed to provoke the crash - there were 6 core dumps, I followed your instructions and have pasted the gdb output below for each one. Let me know if this sheds any light on the problem. Thanks. Patrick -- *# gdb /usr/sbin/httpd /tmp/core.1221* Thread 2 (Thread 0x7f5876bff840 (LWP 1221)): #0 0x7f587518e687 in kill () from /lib64/libc.so.6 #1 #2 0x7f5861def0d8 in ?? () from /lib64/libgcc_s.so.1 #3 0x7f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1 #4 0x7f587526cbf6 in backtrace () from /lib64/libc.so.6 #5 0x7f58751d0fb4 in __libc_message () from /lib64/libc.so.6 #6 0x7f58751d8854 in malloc_consolidate () from /lib64/libc.so.6 #7 0x7f58751d923e in _int_free () from /lib64/libc.so.6 #8 0x7f5875963ded in apr_allocator_destroy () from /usr/lib64/libapr-1.so.0 #9 0x7f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so #10 0x7f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so #11 #12 0x7f587524bcfd in poll () from /lib64/libc.so.6 #13 0x7f5868086682 in ?? () from /etc/httpd/modules/libphp-7.0.so #14 0x7f5867f8fb8b in ?? () from /etc/httpd/modules/libphp-7.0.so #15 0x7f586807ab3e in _php_stream_fill_read_buffer () from /etc/httpd/modules/libphp-7.0.so #16 0x7f586807afbb in _php_stream_get_line () from /etc/httpd/modules/ libphp-7.0.so #17 0x7f586805012f in ?? () from /etc/httpd/modules/libphp-7.0.so #18 0x7f5868052148 in ?? () from /etc/httpd/modules/libphp-7.0.so #19 0x7f586807d079 in _php_stream_open_wrapper_ex () from /etc/httpd/modules/libphp-7.0.so #20 0x7f586801181b in ?? () from /etc/httpd/modules/libphp-7.0.so #21 0x7f585e957b3c in ?? () #22 0x7f5866fcde00 in ?? () #23 0x1c07 in ?? () #24 0x7f585605b380 in ?? () #25 0x7f585605b3a0 in ?? () #26 0x7f5854859bd8 in ?? () #27 0x7f5866fefe00 in ?? () #28 0x7f5853291dc0 in ?? () #29 0x7f586805cf01 in ?? () from /etc/httpd/modules/libphp-7.0.so #30 0x7f586805f6f4 in ?? () from /etc/httpd/modules/libphp-7.0.so #31 0x7f586810c57d in ?? () from /etc/httpd/modules/libphp-7.0.so #32 0x7f58680fe9eb in execute_ex () from /etc/httpd/modules/ libphp-7.0.so #33 0x7f586814849f in zend_execute () from /etc/httpd/modules/ libphp-7.0.so #34 0x7f58680c1e84 in zend_execute_scripts () from /etc/httpd/modules/ libphp-7.0.so #35 0x7f5868064808 in php_execute_script () from /etc/httpd/modules/ libphp-7.0.so #36 0x7f5868149dea in ?? () from /etc/httpd/modules/libphp-7.0.so #37 0x55e5af703bc0 in ap_run_handler () #38 0x55e5af704109 in ap_invoke_handler () #39 0x55e5af71a3ea in ap_process_async_request () #40 0x55e5af71a6be in ap_process_request () #41 0x55e5af716764 in ?? () #42 0x55e5af70d810 in ap_run_process_connection () #43 0x7f586c276b89 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so #44 0x7f586c276dc0 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so #45 0x7f586c277bc6 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so #46 0x55e5af6e3e8e in ap_run_mpm () #47 0x55e5af6dc0b9 in main () Thread 1 (Thread 0x7f5853d61700 (LWP 1222)): #0 0x7f587524da63 in select () from /lib64/libc.so.6 #1 0x7f587596ff65 in apr_sleep () from /usr/lib64/libapr-1.so.0 #2 0x7f586d1f8e62 in ?? () from /etc/httpd/modules/mod_watchdog.so #3 0x7f5875731e75 in start_thread () from /lib64/libpthread.so.0 #4 0x7f5875256a2d in clone () from /lib64/libc.so.6 *# gdb /usr/sbin/httpd /tmp/core.1957* Thread 1 (Thread 0x7f5876bff840 (LWP 1957)): #0 0x7f5861def0d8 in ?? () from /lib64/libgcc_s.so.1 #1 0x7f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1 #2 0x7f587526cbf6 in backtrace () from /lib64/libc.so.6 #3 0x7f58751d0fb4 in __libc_message () from /lib64/libc.so.6 #4 0x7f58751d8854 in malloc_consolidate () from /lib64/libc.so.6 #5 0x7f58751d923e in _int_free () from /lib64/libc.so.6 #6 0x7f5875963ded in apr_allocator_destroy () from /usr/lib64/libapr-1.so.0 #7 0x7f586c276477 in ?? () from /etc/httpd/modules/mod_mpm_prefork.so #8 0x7f586c2764cb in ?? () from /etc/httpd/modules/mod_mpm_prefork.so #9 #10 0x7f587524bcfd in poll () from /lib64/libc.so.6 #11 0x7f5865f45ea9 in ?? () #12 0x55e500d80001 in ?? () #13 0x0006adf5 in ?? () #14 0x000bf117 in ?? () #15 0x00012ea5bfb4 in ?? () #16 0x in ?? () *# gdb /usr/sbin/httpd /tmp/core.29826* Thread 1 (Thread 0x7f5876bff840 (LWP 29826)): #0 0x7f5861def0d8 in ?? () from /lib64/libgcc_s.so.1 #1 0x7f5861deffd9 in _Unwind_Backtrace () from /lib64/libgcc_s.so.1 #2 0x7f587526cbf6 in backtrace () from /lib64/libc.so.6 #3 0x7f58751d0fb4 in __libc_message () from /lib64/libc.so.6 #4 0x7f58751d8854 in malloc_consolidate () from /lib64/libc.so.6 #5 0x7f58751d923e in _int_free () from /lib64/libc.so.6 #6 0x7f5875963ded in apr_allocator_destroy ()
Re: [users@httpd] Issue with Apache 2.4.51 hanging
We will then have to look into what is happening in the step (probably add debugging code): Warm-up cache process kicks off with 2 long running requests (45 seconds each). This is a PHP application running under mod_php - each process grows up to 700 MB, so the application kills the httpd child process at the end to release the memory, using posix_kill(PID, 28). Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Fri, Oct 22, 2021 at 3:07 PM Patrick Verdon wrote: > Correct. > > > On Fri, 22 Oct 2021 at 10:35, Deepak Goel wrote: > >> I guess what you are saying is that the following error happens during >> startup and not during normal operation >> >> ( [Sun Oct 17 15:53:49.244527 2021] [mpm_prefork:error] [pid 3581] >> AH00161: server reached MaxRequestWorkers setting, consider raising the >> MaxRequestWorkers setting) >> >> >> Deepak >> "The greatness of a nation can be judged by the way its animals are >> treated - Mahatma Gandhi" >> >> +91 73500 12833 >> deic...@gmail.com >> >> Facebook: https://www.facebook.com/deicool >> LinkedIn: www.linkedin.com/in/deicool >> >> "Plant a Tree, Go Green" >> >> Make In India : http://www.makeinindia.com/home >> >> >> On Fri, Oct 22, 2021 at 2:23 PM Patrick Verdon < >> patrick.ver...@youreko.com> wrote: >> >>> Hi Yann, >>> >>> Quick update - we've enabled the core dumps but haven't been able to >>> reproduce the issue. After removing mod_http2 the first time we were able >>> to trigger the crash after 14 attempts but we've since tried over 100 times >>> with no luck. We'll keep trying as there's nothing worse than knowing >>> there's a bug lurking that can cause a crash. >>> >>> @Deepak - thanks for your suggestion but hitting MaxRequestWorkers is a >>> quirk of our installation, we load the max workers on startup so that the >>> PHP application is primed and ready, rather than have Apache spawn lots of >>> heavy processes. This is the same configuration we've had for years without >>> ever experiencing Apache hanging until the upgrade to 2.4.48. >>> >>> Thanks. >>> >>> Patrick >>> >>> *--* >>> >>> *Patrick Verdon | Founder* >>> Web: www.youreko.com >>> Mobile: +44 (0)7809 296438 >>> Skype: patrick_verdon >>> >>> This entire communication is sent on behalf of >>> Youreko Ltd and is strictly confidential to and >>> for the sole use of the intended addressee. >>> >>> Registered in England - 7448349 >>> >>> >>> >>> On Tue, 19 Oct 2021 at 11:00, Deepak Goel wrote: >>> Hi Looks like the step 2 in your process is not working in the upgraded version of apache. Therefore it is vomiting out the following: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Mon, Oct 18, 2021 at 2:57 PM Patrick Verdon < patrick.ver...@youreko.com> wrote: > Hi All, > > I'd appreciate some feedback on an issue I'm experiencing. I've spent > quite some time researching the problem as it causes a serious outage in > our application. I've searched the Web, Stack Overflow, this list's mail > archives, the latest Apache bugs, and more, but have not been able to find > any reports of a similar issue. > > Background. I'm running the latest Apache 2.4.51 on Amazon Linux with > mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our > application take about 45 seconds to complete so there is a warm-up cache > procedure at regular intervals during the day which primes the varnish > cache. The following steps reliably cause Apache to hang, requiring a > manual restart: > >1. Varnish cache is cleared, causing spike in load on httpd >2. Warm-up cache process kicks off with 2 long running requests >(45 seconds each). This is a PHP application running under mod_php - > each >process grows up to 700 MB, so the application kills the httpd child >process at the end to release the memory, using posix_kill(PID, 28). >3. Apache hangs and does not recover. Varnish serves 503s. >4. Manual restart required: service httpd restart >5. Errors in the log show that 2 children had segmentation faults, >presumably the 2 with long running processes. > > > Albeit ugly, this process has been running for a year and a half > without any issues. We traced the
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Correct. On Fri, 22 Oct 2021 at 10:35, Deepak Goel wrote: > I guess what you are saying is that the following error happens during > startup and not during normal operation > > ( [Sun Oct 17 15:53:49.244527 2021] [mpm_prefork:error] [pid 3581] > AH00161: server reached MaxRequestWorkers setting, consider raising the > MaxRequestWorkers setting) > > > Deepak > "The greatness of a nation can be judged by the way its animals are > treated - Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Fri, Oct 22, 2021 at 2:23 PM Patrick Verdon > wrote: > >> Hi Yann, >> >> Quick update - we've enabled the core dumps but haven't been able to >> reproduce the issue. After removing mod_http2 the first time we were able >> to trigger the crash after 14 attempts but we've since tried over 100 times >> with no luck. We'll keep trying as there's nothing worse than knowing >> there's a bug lurking that can cause a crash. >> >> @Deepak - thanks for your suggestion but hitting MaxRequestWorkers is a >> quirk of our installation, we load the max workers on startup so that the >> PHP application is primed and ready, rather than have Apache spawn lots of >> heavy processes. This is the same configuration we've had for years without >> ever experiencing Apache hanging until the upgrade to 2.4.48. >> >> Thanks. >> >> Patrick >> >> *--* >> >> *Patrick Verdon | Founder* >> Web: www.youreko.com >> Mobile: +44 (0)7809 296438 >> Skype: patrick_verdon >> >> This entire communication is sent on behalf of >> Youreko Ltd and is strictly confidential to and >> for the sole use of the intended addressee. >> >> Registered in England - 7448349 >> >> >> >> On Tue, 19 Oct 2021 at 11:00, Deepak Goel wrote: >> >>> Hi >>> >>> Looks like the step 2 in your process is not working in the upgraded >>> version of apache. >>> >>> Therefore it is vomiting out the following: >>> server reached MaxRequestWorkers setting, consider raising the >>> MaxRequestWorkers setting >>> >>> Deepak >>> "The greatness of a nation can be judged by the way its animals are >>> treated - Mahatma Gandhi" >>> >>> +91 73500 12833 >>> deic...@gmail.com >>> >>> Facebook: https://www.facebook.com/deicool >>> LinkedIn: www.linkedin.com/in/deicool >>> >>> "Plant a Tree, Go Green" >>> >>> Make In India : http://www.makeinindia.com/home >>> >>> >>> On Mon, Oct 18, 2021 at 2:57 PM Patrick Verdon < >>> patrick.ver...@youreko.com> wrote: >>> Hi All, I'd appreciate some feedback on an issue I'm experiencing. I've spent quite some time researching the problem as it causes a serious outage in our application. I've searched the Web, Stack Overflow, this list's mail archives, the latest Apache bugs, and more, but have not been able to find any reports of a similar issue. Background. I'm running the latest Apache 2.4.51 on Amazon Linux with mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our application take about 45 seconds to complete so there is a warm-up cache procedure at regular intervals during the day which primes the varnish cache. The following steps reliably cause Apache to hang, requiring a manual restart: 1. Varnish cache is cleared, causing spike in load on httpd 2. Warm-up cache process kicks off with 2 long running requests (45 seconds each). This is a PHP application running under mod_php - each process grows up to 700 MB, so the application kills the httpd child process at the end to release the memory, using posix_kill(PID, 28). 3. Apache hangs and does not recover. Varnish serves 503s. 4. Manual restart required: service httpd restart 5. Errors in the log show that 2 children had segmentation faults, presumably the 2 with long running processes. Albeit ugly, this process has been running for a year and a half without any issues. We traced the date that crashes started to the date Apache was upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an issue in 2.4.51. See the error_log below and details about the installation. Any feedback on where to report this issue would be much appreciated. Thanks. Patrick -- # cat /var/log/httpd/error_log httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd:
Re: [users@httpd] Issue with Apache 2.4.51 hanging
I guess what you are saying is that the following error happens during startup and not during normal operation ( [Sun Oct 17 15:53:49.244527 2021] [mpm_prefork:error] [pid 3581] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting) Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Fri, Oct 22, 2021 at 2:23 PM Patrick Verdon wrote: > Hi Yann, > > Quick update - we've enabled the core dumps but haven't been able to > reproduce the issue. After removing mod_http2 the first time we were able > to trigger the crash after 14 attempts but we've since tried over 100 times > with no luck. We'll keep trying as there's nothing worse than knowing > there's a bug lurking that can cause a crash. > > @Deepak - thanks for your suggestion but hitting MaxRequestWorkers is a > quirk of our installation, we load the max workers on startup so that the > PHP application is primed and ready, rather than have Apache spawn lots of > heavy processes. This is the same configuration we've had for years without > ever experiencing Apache hanging until the upgrade to 2.4.48. > > Thanks. > > Patrick > > *--* > > *Patrick Verdon | Founder* > Web: www.youreko.com > Mobile: +44 (0)7809 296438 > Skype: patrick_verdon > > This entire communication is sent on behalf of > Youreko Ltd and is strictly confidential to and > for the sole use of the intended addressee. > > Registered in England - 7448349 > > > > On Tue, 19 Oct 2021 at 11:00, Deepak Goel wrote: > >> Hi >> >> Looks like the step 2 in your process is not working in the upgraded >> version of apache. >> >> Therefore it is vomiting out the following: >> server reached MaxRequestWorkers setting, consider raising the >> MaxRequestWorkers setting >> >> Deepak >> "The greatness of a nation can be judged by the way its animals are >> treated - Mahatma Gandhi" >> >> +91 73500 12833 >> deic...@gmail.com >> >> Facebook: https://www.facebook.com/deicool >> LinkedIn: www.linkedin.com/in/deicool >> >> "Plant a Tree, Go Green" >> >> Make In India : http://www.makeinindia.com/home >> >> >> On Mon, Oct 18, 2021 at 2:57 PM Patrick Verdon < >> patrick.ver...@youreko.com> wrote: >> >>> Hi All, >>> >>> I'd appreciate some feedback on an issue I'm experiencing. I've spent >>> quite some time researching the problem as it causes a serious outage in >>> our application. I've searched the Web, Stack Overflow, this list's mail >>> archives, the latest Apache bugs, and more, but have not been able to find >>> any reports of a similar issue. >>> >>> Background. I'm running the latest Apache 2.4.51 on Amazon Linux with >>> mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our >>> application take about 45 seconds to complete so there is a warm-up cache >>> procedure at regular intervals during the day which primes the varnish >>> cache. The following steps reliably cause Apache to hang, requiring a >>> manual restart: >>> >>>1. Varnish cache is cleared, causing spike in load on httpd >>>2. Warm-up cache process kicks off with 2 long running requests (45 >>>seconds each). This is a PHP application running under mod_php - each >>>process grows up to 700 MB, so the application kills the httpd child >>>process at the end to release the memory, using posix_kill(PID, 28). >>>3. Apache hangs and does not recover. Varnish serves 503s. >>>4. Manual restart required: service httpd restart >>>5. Errors in the log show that 2 children had segmentation faults, >>>presumably the 2 with long running processes. >>> >>> >>> Albeit ugly, this process has been running for a year and a half without >>> any issues. We traced the date that crashes started to the date Apache was >>> upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an >>> issue in 2.4.51. >>> >>> See the error_log below and details about the installation. >>> >>> Any feedback on where to report this issue would be much appreciated. >>> >>> Thanks. >>> >>> Patrick >>> >>> -- >>> >>> # cat /var/log/httpd/error_log >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >>> 0' failed. >>> httpd: misc/apr_reslist.c:161: reslist_cleanup:
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Yann, Quick update - we've enabled the core dumps but haven't been able to reproduce the issue. After removing mod_http2 the first time we were able to trigger the crash after 14 attempts but we've since tried over 100 times with no luck. We'll keep trying as there's nothing worse than knowing there's a bug lurking that can cause a crash. @Deepak - thanks for your suggestion but hitting MaxRequestWorkers is a quirk of our installation, we load the max workers on startup so that the PHP application is primed and ready, rather than have Apache spawn lots of heavy processes. This is the same configuration we've had for years without ever experiencing Apache hanging until the upgrade to 2.4.48. Thanks. Patrick *--* *Patrick Verdon | Founder* Web: www.youreko.com Mobile: +44 (0)7809 296438 Skype: patrick_verdon This entire communication is sent on behalf of Youreko Ltd and is strictly confidential to and for the sole use of the intended addressee. Registered in England - 7448349 On Tue, 19 Oct 2021 at 11:00, Deepak Goel wrote: > Hi > > Looks like the step 2 in your process is not working in the upgraded > version of apache. > > Therefore it is vomiting out the following: > server reached MaxRequestWorkers setting, consider raising the > MaxRequestWorkers setting > > Deepak > "The greatness of a nation can be judged by the way its animals are > treated - Mahatma Gandhi" > > +91 73500 12833 > deic...@gmail.com > > Facebook: https://www.facebook.com/deicool > LinkedIn: www.linkedin.com/in/deicool > > "Plant a Tree, Go Green" > > Make In India : http://www.makeinindia.com/home > > > On Mon, Oct 18, 2021 at 2:57 PM Patrick Verdon > wrote: > >> Hi All, >> >> I'd appreciate some feedback on an issue I'm experiencing. I've spent >> quite some time researching the problem as it causes a serious outage in >> our application. I've searched the Web, Stack Overflow, this list's mail >> archives, the latest Apache bugs, and more, but have not been able to find >> any reports of a similar issue. >> >> Background. I'm running the latest Apache 2.4.51 on Amazon Linux with >> mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our >> application take about 45 seconds to complete so there is a warm-up cache >> procedure at regular intervals during the day which primes the varnish >> cache. The following steps reliably cause Apache to hang, requiring a >> manual restart: >> >>1. Varnish cache is cleared, causing spike in load on httpd >>2. Warm-up cache process kicks off with 2 long running requests (45 >>seconds each). This is a PHP application running under mod_php - each >>process grows up to 700 MB, so the application kills the httpd child >>process at the end to release the memory, using posix_kill(PID, 28). >>3. Apache hangs and does not recover. Varnish serves 503s. >>4. Manual restart required: service httpd restart >>5. Errors in the log show that 2 children had segmentation faults, >>presumably the 2 with long running processes. >> >> >> Albeit ugly, this process has been running for a year and a half without >> any issues. We traced the date that crashes started to the date Apache was >> upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an >> issue in 2.4.51. >> >> See the error_log below and details about the installation. >> >> Any feedback on where to report this issue would be much appreciated. >> >> Thanks. >> >> Patrick >> >> -- >> >> # cat /var/log/httpd/error_log >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == >> 0' failed. >> httpd: misc/apr_reslist.c:161:
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Looks like the step 2 in your process is not working in the upgraded version of apache. Therefore it is vomiting out the following: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting Deepak "The greatness of a nation can be judged by the way its animals are treated - Mahatma Gandhi" +91 73500 12833 deic...@gmail.com Facebook: https://www.facebook.com/deicool LinkedIn: www.linkedin.com/in/deicool "Plant a Tree, Go Green" Make In India : http://www.makeinindia.com/home On Mon, Oct 18, 2021 at 2:57 PM Patrick Verdon wrote: > Hi All, > > I'd appreciate some feedback on an issue I'm experiencing. I've spent > quite some time researching the problem as it causes a serious outage in > our application. I've searched the Web, Stack Overflow, this list's mail > archives, the latest Apache bugs, and more, but have not been able to find > any reports of a similar issue. > > Background. I'm running the latest Apache 2.4.51 on Amazon Linux with > mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our > application take about 45 seconds to complete so there is a warm-up cache > procedure at regular intervals during the day which primes the varnish > cache. The following steps reliably cause Apache to hang, requiring a > manual restart: > >1. Varnish cache is cleared, causing spike in load on httpd >2. Warm-up cache process kicks off with 2 long running requests (45 >seconds each). This is a PHP application running under mod_php - each >process grows up to 700 MB, so the application kills the httpd child >process at the end to release the memory, using posix_kill(PID, 28). >3. Apache hangs and does not recover. Varnish serves 503s. >4. Manual restart required: service httpd restart >5. Errors in the log show that 2 children had segmentation faults, >presumably the 2 with long running processes. > > > Albeit ugly, this process has been running for a year and a half without > any issues. We traced the date that crashes started to the date Apache was > upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an > issue in 2.4.51. > > See the error_log below and details about the installation. > > Any feedback on where to report this issue would be much appreciated. > > Thanks. > > Patrick > > -- > > # cat /var/log/httpd/error_log > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: child > pid 3166 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990531 2021] [core:notice] [pid 2620] AH00052: child > pid 3483 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990545 2021] [core:notice] [pid 2620] AH00052: child > pid 2657 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990557 2021] [core:notice] [pid 2620] AH00052: child > pid 2660 exit signal Aborted (6) > [Sun Oct 17 15:53:47.990568 2021] [core:notice] [pid 2620] AH00052:
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Patrick, On Mon, Oct 18, 2021 at 10:13 PM Patrick Verdon wrote: > > Just a quick follow up - we've tried removing mod_http2 but still managed to > provoke a crash. See the error_log below when stopping/restarting after httpd > becomes unresponsive. It seems to have eliminated the "reslist_cleanup: Assertion `rl->ntotal == 0' failed" and "Aborted (6)" errors, which was the primary goal. Hopefully the other "corrupted size vs. prev_size" and "Segmentation fault (11)" errors were related but it does not seem to be the case.. > We need to be a bit more careful removing other modules to make sure they're > not used, which is more time consuming - do you think this is still worth > doing to address the issue? I can't tell this from the few pieces of information available so far. > > If you have any other suggestions let me know. Since httpd is now crashing with "Segmentation fault" (only), there is a way to get a coredump file generated for further analysis, you need to add this to your main/base httpd configuration: CoreDumpDirectory /tmp After each crash there should be a "/tmp/core" (or "/tmp/core.[pid]") file which can be analysed with the gdb debugger, by using these commands: $ gdb /usr/sbin/httpd /tmp/core[.pid] [and once in gdb with the "(gdb)" prompt] (gdb) thread apply all bt Please paste the result here. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Yann, Just a quick follow up - we've tried removing mod_http2 but still managed to provoke a crash. See the error_log below when stopping/restarting after httpd becomes unresponsive. We need to be a bit more careful removing other modules to make sure they're not used, which is more time consuming - do you think this is still worth doing to address the issue? If you have any other suggestions let me know. Thanks. Patrick -- # cat /var/log/httpd/error_log *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x55a67cc31e7f *** *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x55a67cc31e7f *** *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x55a67cc31e7f *** *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x55a67cc31e7f *** [Mon Oct 18 20:59:48.426225 2021] [core:notice] [pid 31207] AH00052: child pid 32036 exit signal Segmentation fault (11) [Mon Oct 18 20:59:48.426389 2021] [core:notice] [pid 31207] AH00052: child pid 31246 exit signal Segmentation fault (11) [Mon Oct 18 20:59:48.492282 2021] [core:notice] [pid 31207] AH00052: child pid 31253 exit signal Segmentation fault (11) [Mon Oct 18 20:59:48.492312 2021] [core:notice] [pid 31207] AH00052: child pid 32289 exit signal Segmentation fault (11) [Mon Oct 18 20:59:48.492455 2021] [mpm_prefork:notice] [pid 31207] AH00169: caught SIGTERM, shutting down [Mon Oct 18 20:59:48.631928 2021] [suexec:notice] [pid 32620] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Mon Oct 18 20:59:48.662384 2021] [lbmethod_heartbeat:notice] [pid 32626] AH02282: No slotmem from mod_heartmonitor [Mon Oct 18 20:59:48.724408 2021] [mpm_prefork:notice] [pid 32626] AH00163: Apache/2.4.51 (Amazon) OpenSSL/1.0.2k-fips configured -- resuming normal operations [Mon Oct 18 20:59:48.724430 2021] [core:notice] [pid 32626] AH00094: Command line: '/usr/sbin/httpd' [Mon Oct 18 20:59:49.724509 2021] [mpm_prefork:error] [pid 32626] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting *--* *Patrick Verdon | Founder* Web: www.youreko.com Mobile: +44 (0)7809 296438 Skype: patrick_verdon This entire communication is sent on behalf of Youreko Ltd and is strictly confidential to and for the sole use of the intended addressee. Registered in England - 7448349 On Mon, 18 Oct 2021 at 15:05, Patrick Verdon wrote: > Hi Yann, > > Many thanks for the super quick response. We'll try to remove mod_http2 > and other modules as you suggest to see if that helps. I'll get back to you > once we've had a chance to test it. > > Thanks. > > Patrick > > *--* > > *Patrick Verdon | Founder* > Web: www.youreko.com > Mobile: +44 (0)7809 296438 > Skype: patrick_verdon > > This entire communication is sent on behalf of > Youreko Ltd and is strictly confidential to and > for the sole use of the intended addressee. > > Registered in England - 7448349 > > > > On Mon, 18 Oct 2021 at 12:57, Yann Ylavic wrote: > >> Hi Patrick, >> >> On Mon, Oct 18, 2021 at 11:27 AM Patrick Verdon >> wrote: >> > >> > # cat /var/log/httpd/error_log >> > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal >> == 0' failed. >> [] >> > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: >> 0x557f94567e4f *** >> [] >> > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal >> == 0' failed. >> > [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: >> child pid 3166 exit signal Aborted (6) >> [] >> > [Sun Oct 17 15:53:47.990781 2021] [core:notice] [pid 2620] AH00052: >> child pid 2741 exit signal Segmentation fault (11) >> > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: >> 0x557f94567e4f *** >> [] >> > [Sun Oct 17 15:53:48.056599 2021] [core:notice] [pid 2620] AH00052: >> child pid 2727 exit signal Aborted (6) >> > [Sun Oct 17 15:53:48.056667 2021] [mpm_prefork:notice] [pid 2620] >> AH00169: caught SIGTERM, shutting down >> >> The log seems to show a stop then start sequence (which is possibly >> what "service httpd restart" does), anyway the stop crashes children >> processes that at some point have reserved/handled mod_proxy >> connections. >> >> We will discuss whether/how to fix this on the dev@ mailing list, in >> the meantime I'd suggest that: >> >> > [Sun Oct 17 15:53:48.180621 2021] [http2:warn] [pid 3581] AH10034: The >> mpm module (prefork.c) is not supported by mod_http2. The mpm determines >> how things are processed in your server. HTTP/2 has more demands in this >> regard and the currently selected mpm will just not do. This is an advisory >> warning. Your server will continue to work, but the HTTP/2 protocol will be >> inactive. >> >> .. you do not "LoadModule http2_module mod_http2.so" in your MPM >> prefork configuration, because due to its multithreaded nature (unlike >> MPM prefork) mod_http2 implies that mod_proxy will have to >> allocate/handle multiple simultaneous connection to the backend which >> is
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Yann, Many thanks for the super quick response. We'll try to remove mod_http2 and other modules as you suggest to see if that helps. I'll get back to you once we've had a chance to test it. Thanks. Patrick *--* *Patrick Verdon | Founder* Web: www.youreko.com Mobile: +44 (0)7809 296438 Skype: patrick_verdon This entire communication is sent on behalf of Youreko Ltd and is strictly confidential to and for the sole use of the intended addressee. Registered in England - 7448349 On Mon, 18 Oct 2021 at 12:57, Yann Ylavic wrote: > Hi Patrick, > > On Mon, Oct 18, 2021 at 11:27 AM Patrick Verdon > wrote: > > > > # cat /var/log/httpd/error_log > > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > [] > > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** > [] > > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == > 0' failed. > > [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: > child pid 3166 exit signal Aborted (6) > [] > > [Sun Oct 17 15:53:47.990781 2021] [core:notice] [pid 2620] AH00052: > child pid 2741 exit signal Segmentation fault (11) > > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** > [] > > [Sun Oct 17 15:53:48.056599 2021] [core:notice] [pid 2620] AH00052: > child pid 2727 exit signal Aborted (6) > > [Sun Oct 17 15:53:48.056667 2021] [mpm_prefork:notice] [pid 2620] > AH00169: caught SIGTERM, shutting down > > The log seems to show a stop then start sequence (which is possibly > what "service httpd restart" does), anyway the stop crashes children > processes that at some point have reserved/handled mod_proxy > connections. > > We will discuss whether/how to fix this on the dev@ mailing list, in > the meantime I'd suggest that: > > > [Sun Oct 17 15:53:48.180621 2021] [http2:warn] [pid 3581] AH10034: The > mpm module (prefork.c) is not supported by mod_http2. The mpm determines > how things are processed in your server. HTTP/2 has more demands in this > regard and the currently selected mpm will just not do. This is an advisory > warning. Your server will continue to work, but the HTTP/2 protocol will be > inactive. > > .. you do not "LoadModule http2_module mod_http2.so" in your MPM > prefork configuration, because due to its multithreaded nature (unlike > MPM prefork) mod_http2 implies that mod_proxy will have to > allocate/handle multiple simultaneous connection to the backend which > is what is causing the crash here. > > > [Sun Oct 17 15:53:48.181146 2021] [lbmethod_heartbeat:notice] [pid 3581] > AH02282: No slotmem from mod_heartmonitor > > Likewise you probably don't need lbmethod_heartbeat and several > modules in your list, so I'd suggest that you cleanup your LoadModules > a bit, ideally to the strict minimum needed. > > > Regards; > Yann. > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > >
Re: [users@httpd] Issue with Apache 2.4.51 hanging
Hi Patrick, On Mon, Oct 18, 2021 at 11:27 AM Patrick Verdon wrote: > > # cat /var/log/httpd/error_log > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. [] > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** [] > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. > [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: child pid > 3166 exit signal Aborted (6) [] > [Sun Oct 17 15:53:47.990781 2021] [core:notice] [pid 2620] AH00052: child pid > 2741 exit signal Segmentation fault (11) > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** [] > [Sun Oct 17 15:53:48.056599 2021] [core:notice] [pid 2620] AH00052: child pid > 2727 exit signal Aborted (6) > [Sun Oct 17 15:53:48.056667 2021] [mpm_prefork:notice] [pid 2620] AH00169: > caught SIGTERM, shutting down The log seems to show a stop then start sequence (which is possibly what "service httpd restart" does), anyway the stop crashes children processes that at some point have reserved/handled mod_proxy connections. We will discuss whether/how to fix this on the dev@ mailing list, in the meantime I'd suggest that: > [Sun Oct 17 15:53:48.180621 2021] [http2:warn] [pid 3581] AH10034: The mpm > module (prefork.c) is not supported by mod_http2. The mpm determines how > things are processed in your server. HTTP/2 has more demands in this regard > and the currently selected mpm will just not do. This is an advisory warning. > Your server will continue to work, but the HTTP/2 protocol will be inactive. .. you do not "LoadModule http2_module mod_http2.so" in your MPM prefork configuration, because due to its multithreaded nature (unlike MPM prefork) mod_http2 implies that mod_proxy will have to allocate/handle multiple simultaneous connection to the backend which is what is causing the crash here. > [Sun Oct 17 15:53:48.181146 2021] [lbmethod_heartbeat:notice] [pid 3581] > AH02282: No slotmem from mod_heartmonitor Likewise you probably don't need lbmethod_heartbeat and several modules in your list, so I'd suggest that you cleanup your LoadModules a bit, ideally to the strict minimum needed. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
[users@httpd] Issue with Apache 2.4.51 hanging
Hi All, I'd appreciate some feedback on an issue I'm experiencing. I've spent quite some time researching the problem as it causes a serious outage in our application. I've searched the Web, Stack Overflow, this list's mail archives, the latest Apache bugs, and more, but have not been able to find any reports of a similar issue. Background. I'm running the latest Apache 2.4.51 on Amazon Linux with mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our application take about 45 seconds to complete so there is a warm-up cache procedure at regular intervals during the day which primes the varnish cache. The following steps reliably cause Apache to hang, requiring a manual restart: 1. Varnish cache is cleared, causing spike in load on httpd 2. Warm-up cache process kicks off with 2 long running requests (45 seconds each). This is a PHP application running under mod_php - each process grows up to 700 MB, so the application kills the httpd child process at the end to release the memory, using posix_kill(PID, 28). 3. Apache hangs and does not recover. Varnish serves 503s. 4. Manual restart required: service httpd restart 5. Errors in the log show that 2 children had segmentation faults, presumably the 2 with long running processes. Albeit ugly, this process has been running for a year and a half without any issues. We traced the date that crashes started to the date Apache was upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an issue in 2.4.51. See the error_log below and details about the installation. Any feedback on where to report this issue would be much appreciated. Thanks. Patrick -- # cat /var/log/httpd/error_log httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: 0x557f94567e4f *** httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' failed. [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: child pid 3166 exit signal Aborted (6) [Sun Oct 17 15:53:47.990531 2021] [core:notice] [pid 2620] AH00052: child pid 3483 exit signal Aborted (6) [Sun Oct 17 15:53:47.990545 2021] [core:notice] [pid 2620] AH00052: child pid 2657 exit signal Aborted (6) [Sun Oct 17 15:53:47.990557 2021] [core:notice] [pid 2620] AH00052: child pid 2660 exit signal Aborted (6) [Sun Oct 17 15:53:47.990568 2021] [core:notice] [pid 2620] AH00052: child pid 2661 exit signal Aborted (6) [Sun Oct 17 15:53:47.990579 2021] [core:notice] [pid 2620] AH00052: child pid 3172 exit signal Aborted (6) [Sun Oct 17 15:53:47.990592 2021] [core:notice] [pid 2620] AH00052: child pid 2681 exit signal Aborted (6) [Sun Oct 17 15:53:47.990603 2021] [core:notice] [pid 2620] AH00052: child pid 3254 exit signal Aborted (6) [Sun Oct 17 15:53:47.990615 2021] [core:notice] [pid 2620] AH00052: child pid 2685 exit signal Aborted (6) [Sun Oct 17 15:53:47.990627 2021] [core:notice] [pid 2620] AH00052: child pid 2688 exit signal Aborted (6) [Sun Oct 17 15:53:47.990639 2021] [core:notice] [pid 2620] AH00052: child pid 3015 exit signal Aborted (6) [Sun Oct 17 15:53:47.990652 2021] [core:notice] [pid 2620] AH00052: child pid 2696
