Re: [users@httpd] Self built httpd 2.4.43 problems
Wow, that's it, actually we moved from mpm worker to mpm event on this test machine, and here are the results! I'll check for a more recent APR ;) Thanks a lot! Gabriele Sonicle S.r.l. : http://www.sonicle.com Music: http://www.gabrielebulfon.com eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets -- Da: Yann Ylavic A: users@httpd.apache.org Data: 10 febbraio 2021 0.25.58 CET Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems Hi Gabriele, There have been some fixes to the APR library since 1.5.2, notably on the Solaris apr_pollset_poll() implementation, though I can only think of a bug [1] pertaining to MPM event (not MPM worker which you seem to be using). It could be worth upgrading to a more recent APR-1.6.5 or APR-1.7.0 still. Regards; Yann. [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=61786 On Tue, Feb 9, 2021 at 11:07 AM Gabriele Bulfon wrote: > > Hi, I finally could produce the httpd blocked problem and ran a script to > dump stack of all running processes/threads. > They strangely look all quite the same! I attach here all the httpd threads > dumps, maybe you can help us see why it was blocked? > > Thanks! > Gabriele > > > Sonicle S.r.l. : http://www.sonicle.com > Music: http://www.gabrielebulfon.com > eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets > > > > > -- > > Da: Rainer Canavan > A: users@httpd.apache.org > Data: 2 novembre 2020 17.19.27 CET > Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems > > On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon > wrote: > > > > Thanks, I configured and ran server-status after stopping/starting apache. > > Top output is: > > > [...] > > > What should I check? > > Also, when system blocks I won't be able to see server-status, as it will > > be not responding. > > Should I check it daily and look for a specific info that grows? > > "requests currently being processed" would probably increase if > threads are permanently blocked. I would recommend logging this every > few seconds, so that you can at least check after the fact how quickly > the system filled up. > > You should have ExtendedStatus enabled, which should give you a > complete list of all threads and their states. Any that are active > (probably "W", definitively not "." or "_") processing a single > request for extended periods are suspicious, especially if multiple of > the same kind strat piling up. > > If httpd does not respond to requests anymore, and you have multiple > worker childs, you can sometimes get away with killing one and try to > squeeze a status request in there before it gets overrun again. > > Anyway, serverstatus will only provide rough hints of what's going on. > If it is indeed httpd, you'll probably need gdb backtraces. > > rainer > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > > > > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Hi Gabriele, There have been some fixes to the APR library since 1.5.2, notably on the Solaris apr_pollset_poll() implementation, though I can only think of a bug [1] pertaining to MPM event (not MPM worker which you seem to be using). It could be worth upgrading to a more recent APR-1.6.5 or APR-1.7.0 still. Regards; Yann. [1] https://bz.apache.org/bugzilla/show_bug.cgi?id=61786 On Tue, Feb 9, 2021 at 11:07 AM Gabriele Bulfon wrote: > > Hi, I finally could produce the httpd blocked problem and ran a script to > dump stack of all running processes/threads. > They strangely look all quite the same! I attach here all the httpd threads > dumps, maybe you can help us see why it was blocked? > > Thanks! > Gabriele > > > Sonicle S.r.l. : http://www.sonicle.com > Music: http://www.gabrielebulfon.com > eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets > > > > > -- > > Da: Rainer Canavan > A: users@httpd.apache.org > Data: 2 novembre 2020 17.19.27 CET > Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems > > On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon > wrote: > > > > Thanks, I configured and ran server-status after stopping/starting apache. > > Top output is: > > > [...] > > > What should I check? > > Also, when system blocks I won't be able to see server-status, as it will > > be not responding. > > Should I check it daily and look for a specific info that grows? > > "requests currently being processed" would probably increase if > threads are permanently blocked. I would recommend logging this every > few seconds, so that you can at least check after the fact how quickly > the system filled up. > > You should have ExtendedStatus enabled, which should give you a > complete list of all threads and their states. Any that are active > (probably "W", definitively not "." or "_") processing a single > request for extended periods are suspicious, especially if multiple of > the same kind strat piling up. > > If httpd does not respond to requests anymore, and you have multiple > worker childs, you can sometimes get away with killing one and try to > squeeze a status request in there before it gets overrun again. > > Anyway, serverstatus will only provide rough hints of what's going on. > If it is indeed httpd, you'll probably need gdb backtraces. > > rainer > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org > > > > > - > To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org > For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Hi, I finally could produce the httpd blocked problem and ran a script to dump stack of all running processes/threads. They strangely look all quite the same! I attach here all the httpd threads dumps, maybe you can help us see why it was blocked? Thanks! Gabriele Sonicle S.r.l. : http://www.sonicle.com Music: http://www.gabrielebulfon.com eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets -- Da: Rainer Canavan A: users@httpd.apache.org Data: 2 novembre 2020 17.19.27 CET Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon wrote: > > Thanks, I configured and ran server-status after stopping/starting apache. > Top output is: > [...] > What should I check? > Also, when system blocks I won't be able to see server-status, as it will be > not responding. > Should I check it daily and look for a specific info that grows? "requests currently being processed" would probably increase if threads are permanently blocked. I would recommend logging this every few seconds, so that you can at least check after the fact how quickly the system filled up. You should have ExtendedStatus enabled, which should give you a complete list of all threads and their states. Any that are active (probably "W", definitively not "." or "_") processing a single request for extended periods are suspicious, especially if multiple of the same kind strat piling up. If httpd does not respond to requests anymore, and you have multiple worker childs, you can sometimes get away with killing one and try to squeeze a status request in there before it gets overrun again. Anyway, serverstatus will only provide rough hints of what's going on. If it is indeed httpd, you'll probably need gdb backtraces. rainer - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org apache-stack-traces.tgz Description: application/gnutar - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
On Mon, Nov 30, 2020 at 9:08 AM Gabriele Bulfon wrote: > > Is there any way I can provoke a core dump from the httpd threads when the > freeze happens? > I tried using truss (strace solaris like) but nothing was moving on the httpd > processes. > Analyzing server-status I see nothing strange, and then it sill happens every > 7-10 days... That is usually OS specific. For example gcore, gencore, pstack, or attaching with gdb. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Is there any way I can provoke a core dump from the httpd threads when the freeze happens? I tried using truss (strace solaris like) but nothing was moving on the httpd processes. Analyzing server-status I see nothing strange, and then it sill happens every 7-10 days... Gabriele Sonicle S.r.l. : http://www.sonicle.com Music: http://www.gabrielebulfon.com eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets -- Da: Rainer Canavan A: users@httpd.apache.org Data: 2 novembre 2020 17.19.27 CET Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon wrote: > > Thanks, I configured and ran server-status after stopping/starting apache. > Top output is: > [...] > What should I check? > Also, when system blocks I won't be able to see server-status, as it will be > not responding. > Should I check it daily and look for a specific info that grows? "requests currently being processed" would probably increase if threads are permanently blocked. I would recommend logging this every few seconds, so that you can at least check after the fact how quickly the system filled up. You should have ExtendedStatus enabled, which should give you a complete list of all threads and their states. Any that are active (probably "W", definitively not "." or "_") processing a single request for extended periods are suspicious, especially if multiple of the same kind strat piling up. If httpd does not respond to requests anymore, and you have multiple worker childs, you can sometimes get away with killing one and try to squeeze a status request in there before it gets overrun again. Anyway, serverstatus will only provide rough hints of what's going on. If it is indeed httpd, you'll probably need gdb backtraces. rainer - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
On Mon, Nov 2, 2020 at 4:17 PM Gabriele Bulfon wrote: > > Thanks, I configured and ran server-status after stopping/starting apache. > Top output is: > [...] > What should I check? > Also, when system blocks I won't be able to see server-status, as it will be > not responding. > Should I check it daily and look for a specific info that grows? "requests currently being processed" would probably increase if threads are permanently blocked. I would recommend logging this every few seconds, so that you can at least check after the fact how quickly the system filled up. You should have ExtendedStatus enabled, which should give you a complete list of all threads and their states. Any that are active (probably "W", definitively not "." or "_") processing a single request for extended periods are suspicious, especially if multiple of the same kind strat piling up. If httpd does not respond to requests anymore, and you have multiple worker childs, you can sometimes get away with killing one and try to squeeze a status request in there before it gets overrun again. Anyway, serverstatus will only provide rough hints of what's going on. If it is indeed httpd, you'll probably need gdb backtraces. rainer - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
Thanks, I configured and ran server-status after stopping/starting apache. Top output is: Current Time: Monday, 02-Nov-2020 16:16:14 CETRestart Time: Monday, 02-Nov-2020 16:04:54 CETParent Server Config. Generation: 1Parent Server MPM Generation: 0Server uptime: 11 minutes 20 secondsServer load: 0.16 0.23 0.35Total accesses: 553 - Total Traffic: 10.6 MB - Total Duration: 1637098CPU Usage: u4.22 s.41 cu0 cs0 - .681% CPU load.813 requests/sec - 15.9 kB/second - 19.6 kB/request - 2960.39 ms/request5 requests currently being processed, 95 idle workers What should I check? Also, when system blocks I won't be able to see server-status, as it will be not responding. Should I check it daily and look for a specific info that grows? Gabriele Sonicle S.r.l. : http://www.sonicle.com Music: http://www.gabrielebulfon.com eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets -- Da: Rainer Canavan A: users@httpd.apache.org Data: 2 novembre 2020 14.24.06 CET Oggetto: Re: [users@httpd] Self built httpd 2.4.43 problems On Mon, Nov 2, 2020 at 11:13 AM Gabriele Bulfon wrote: [...] Recently we built version 2.4.43 and installed on a test machine. Here, we are experiencing a problem where almost once a week we have to restart apache, which is no more responding. Threads are there, but none is answering on port 80, waiting forever. [...] What may be the issue? Lots of options, more information needed. If it's an issue that slowly eats up your worker threads, monitoring the server-status page with ExtendedStatus On may provide good hints. Otherwise, wait until it stops responding and try to check with strace if anything suspicious is going on. Finally, check "thread apply all bt" in a gdb session attached to some of the blocked processes, possibly a few times with cont and Ctrl+C inbetween to see where it's really waiting. You may need to re-build with debug info, and/or install the debug info for all libraries your httpd is linked against. rainer - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Self built httpd 2.4.43 problems
On Mon, Nov 2, 2020 at 11:13 AM Gabriele Bulfon wrote: >[...] > Recently we built version 2.4.43 and installed on a test machine. > Here, we are experiencing a problem where almost once a week we have to > restart apache, which is no more responding. > Threads are there, but none is answering on port 80, waiting forever. [...] > What may be the issue? Lots of options, more information needed. If it's an issue that slowly eats up your worker threads, monitoring the server-status page with ExtendedStatus On may provide good hints. Otherwise, wait until it stops responding and try to check with strace if anything suspicious is going on. Finally, check "thread apply all bt" in a gdb session attached to some of the blocked processes, possibly a few times with cont and Ctrl+C inbetween to see where it's really waiting. You may need to re-build with debug info, and/or install the debug info for all libraries your httpd is linked against. rainer - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
[users@httpd] Self built httpd 2.4.43 problems
Hi, as the mantainers of the XStreamOS/illumos distro, we also have specific server packages where we build Apache httpd together with other server components. We have systems with apache 2.4.27 running for years without particular problems. Recently we built version 2.4.43 and installed on a test machine. Here, we are experiencing a problem where almost once a week we have to restart apache, which is no more responding. Threads are there, but none is answering on port 80, waiting forever. Here is the output of httpd -V: Server version: Apache/2.4.43 (Unix) Server built: Jul 2 2020 02:26:34 Server's Module Magic Number: 20120211:92 Server loaded: APR 1.5.2, APR-UTIL 1.5.4 Compiled using: APR 1.5.2, APR-UTIL 1.5.4 Architecture: 32-bit Server MPM: event threaded: yes (fixed thread count) forked: yes (variable process count) Server compiled with -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_PROC_PTHREAD_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=256 -D HTTPD_ROOT="/sonicle" -D SUEXEC_BIN="/sonicle/bin/suexec" -D DEFAULT_PIDLOG="var/apache/logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="etc/apache/mime.types" -D SERVER_CONFIG_FILE="etc/apache/httpd.conf" What may be the issue? Thanks Gabriele Sonicle S.r.l. : http://www.sonicle.com Music: http://www.gabrielebulfon.com eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
