Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-27 Thread Patrick Verdon
Thanks @William for the suggestion. I haven't tried strace, I'll look into it, although I've now experienced the crash without the 2 long running processes - so it seems like it may be just the application under load when the varnish cache is first cleared that causes the problem, which is not

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-27 Thread Yann Ylavic
On Tue, Oct 26, 2021 at 7:36 PM Patrick Verdon wrote: > > Do you know who maintains mod_php, is it worth following up with them? I'd suggest reporting the issue to the php maintainers (https://bugs.php.net/). It may ring a bell there.. Regards; Yann.

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-26 Thread William Edwards
> Op 18 okt. 2021 om 11:27 heeft Patrick Verdon > het volgende geschreven: > >  > Hi All, > > I'd appreciate some feedback on an issue I'm experiencing. I've spent quite > some time researching the problem as it causes a serious outage in our > application. I've searched the Web, Stack

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-26 Thread Patrick Verdon
I appreciate saying that "nothing else has changed" is not particularly helpful - although on reflection, this is actually not quite true, a number of other system updates were applied as well as httpd, listed below. The PHP installation was not updated. Unfortunately I don't have much more to go

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-26 Thread Daniel Ferradal
I think we are deviating a bit offtopic here. but still just a little note: mpm_prefork is not problematic, what is problematic is when people do not size httpd with mpm_prefork to deal with the amount of incoming requests they receive. processes are more expensive to generate than threads and

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-26 Thread Ruben Safir
mpm_prefork has been problematic, FWIW On Tue, Oct 26, 2021 at 11:17:31AM +0100, Patrick Verdon wrote: > Hi Daniel, > > Thanks for the feedback. > > Unfortunately that's not really an option for us, as it's a major > architecture change, and is only something we would consider as a > last

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-26 Thread Patrick Verdon
Hi Daniel, Thanks for the feedback. Unfortunately that's not really an option for us, as it's a major architecture change, and is only something we would consider as a last resort. Using mod_php works well for us and our configuration/application has not changed in years - the crashes started

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-26 Thread Daniel Ferradal
Hello, Seems it is related to third party php module being loaded in Apache HTTPD, no? It would be awesome if you can move your php files to be dealt with php-fpm instead and from apache just proxy to php-fpm.. and see if your Apache server would ever hang again (if you do it you could even use

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-22 Thread Patrick Verdon
Hi Yann, I finally managed to provoke the crash - there were 6 core dumps, I followed your instructions and have pasted the gdb output below for each one. Let me know if this sheds any light on the problem. Thanks. Patrick -- *# gdb /usr/sbin/httpd /tmp/core.1221* Thread 2 (Thread

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-22 Thread Deepak Goel
We will then have to look into what is happening in the step (probably add debugging code): Warm-up cache process kicks off with 2 long running requests (45 seconds each). This is a PHP application running under mod_php - each process grows up to 700 MB, so the application kills the httpd child

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-22 Thread Patrick Verdon
Correct. On Fri, 22 Oct 2021 at 10:35, Deepak Goel wrote: > I guess what you are saying is that the following error happens during > startup and not during normal operation > > ( [Sun Oct 17 15:53:49.244527 2021] [mpm_prefork:error] [pid 3581] > AH00161: server reached MaxRequestWorkers

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-22 Thread Deepak Goel
I guess what you are saying is that the following error happens during startup and not during normal operation ( [Sun Oct 17 15:53:49.244527 2021] [mpm_prefork:error] [pid 3581] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting) Deepak "The

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-22 Thread Patrick Verdon
Hi Yann, Quick update - we've enabled the core dumps but haven't been able to reproduce the issue. After removing mod_http2 the first time we were able to trigger the crash after 14 attempts but we've since tried over 100 times with no luck. We'll keep trying as there's nothing worse than knowing

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-19 Thread Deepak Goel
Hi Looks like the step 2 in your process is not working in the upgraded version of apache. Therefore it is vomiting out the following: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting Deepak "The greatness of a nation can be judged by the way its animals

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-18 Thread Yann Ylavic
Hi Patrick, On Mon, Oct 18, 2021 at 10:13 PM Patrick Verdon wrote: > > Just a quick follow up - we've tried removing mod_http2 but still managed to > provoke a crash. See the error_log below when stopping/restarting after httpd > becomes unresponsive. It seems to have eliminated the

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-18 Thread Patrick Verdon
Hi Yann, Just a quick follow up - we've tried removing mod_http2 but still managed to provoke a crash. See the error_log below when stopping/restarting after httpd becomes unresponsive. We need to be a bit more careful removing other modules to make sure they're not used, which is more time

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-18 Thread Patrick Verdon
Hi Yann, Many thanks for the super quick response. We'll try to remove mod_http2 and other modules as you suggest to see if that helps. I'll get back to you once we've had a chance to test it. Thanks. Patrick *--* *Patrick Verdon | Founder* Web: www.youreko.com Mobile: +44 (0)7809 296438

Re: [users@httpd] Issue with Apache 2.4.51 hanging

2021-10-18 Thread Yann Ylavic
Hi Patrick, On Mon, Oct 18, 2021 at 11:27 AM Patrick Verdon wrote: > > # cat /var/log/httpd/error_log > httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal == 0' > failed. [] > *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size: > 0x557f94567e4f *** [] > httpd:

[users@httpd] Issue with Apache 2.4.51 hanging

2021-10-18 Thread Patrick Verdon
Hi All, I'd appreciate some feedback on an issue I'm experiencing. I've spent quite some time researching the problem as it causes a serious outage in our application. I've searched the Web, Stack Overflow, this list's mail archives, the latest Apache bugs, and more, but have not been able to