Hi Ranier Caravan,
Looks like this problem didn't go away.
To get symbols I had to first:
sudo debuginfo-install httpd
Then
sudo gdb -p PID
Several hundred apache forks are present so I selected random one....
*(gdb) where*
#0 0x00007f247f302f0d in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f247fa1e94b in poll (__timeout=30000, __nfds=1,
__fds=0x7ffd5cd92b00) at /usr/include/bits/poll2.h:46
#2 apr_wait_for_io_or_timeout (f=f@entry=0x0, s=s@entry=0x55a9964b7610,
for_read=for_read@entry=1) at support/unix/waitio.c:51
#3 0x00007f247fa1877a in apr_socket_recv (sock=sock@entry=0x55a9964b7610,
buf=buf@entry=0x55a996541b08
"2\r\nh\n\r\n2\r\nh\n\r\n2\r\nh\n\r\n2\r\nh\n\r\ny:
Express\r\nCache-Control: no-store, no-cache, no-transform,
must-revalidate, max-age=0\r\nAccess-Control-Allow-Origin: *\r\nVary:
Origin\r\nContent-Type: application/javascript;"...,
len=len@entry=0x7ffd5cd92be0)
at network_io/unix/sendrecv.c:87
#4 0x00007f2480457281 in socket_bucket_read (a=0x55a99653dc48,
str=0x7ffd5cd92bd8, len=0x7ffd5cd92be0, block=<optimized out>) at
buckets/apr_buckets_socket.c:36
#5 0x00007f2480455756 in apr_brigade_split_line
(bbOut=bbOut@entry=0x55a9964d8ed8,
bbIn=0x55a9964b7f28, block=block@entry=APR_BLOCK_READ,
maxbytes=maxbytes@entry=8192) at buckets/apr_brigade.c:319
#6 0x000055a993422d5c in ap_core_input_filter (f=0x55a9964b7e28,
b=0x55a9964d8ed8, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
at core_filters.c:147
#7 0x00007f247a134c4e in logio_in_filter (f=<optimized out>,
bb=0x55a9964d8ed8, mode=<optimized out>, block=<optimized out>,
readbytes=<optimized out>) at mod_logio.c:140
#8 0x000055a99343e2bd in ap_http_filter (f=<optimized out>, b=<optimized
out>, mode=<optimized out>, block=<optimized out>, readbytes=8192) at
http_filters.c:515
#9 0x00007f2474eed058 in ap_proxy_http_process_response
(p=p@entry=0x55a9965018e8,
r=r@entry=0x55a996501960, backend_ptr=backend_ptr@entry=0x7ffd5cd97058,
server_portstr=server_portstr@entry=0x7ffd5cd97100 "", conf=0x55a994beaef0,
conf=0x55a994beaef0, worker=<optimized out>) at mod_proxy_http.c:1708
*#10 0x00007f2474eef7f9 in proxy_http_handler (r=<optimized out>,
worker=<optimized out>, conf=0x55a994beaef0, url=0x55a9964d819e
"http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming
<http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming>",
*
* proxyname=0x0, proxyport=0) at mod_proxy_http.c:2038*
*#11 0x00007f247673edc4 in proxy_run_scheme_handler
(r=r@entry=0x55a996501960, worker=0x55a994beb1b0,
conf=conf@entry=0x55a994beaef0, *
* url=0x55a9964d819e
"http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming
<http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming>",
proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at
mod_proxy.c:2746*
#12 0x00007f247673fcad in proxy_handler (r=0x55a996501960) at
mod_proxy.c:1125
#13 0x000055a993427990 in ap_run_handler (r=r@entry=0x55a996501960) at
config.c:169
#14 0x000055a993427ed9 in ap_invoke_handler (r=r@entry=0x55a996501960) at
config.c:439
#15 0x000055a99343ca8a in ap_process_async_request (r=r@entry=0x55a996501960)
at http_request.c:328
#16 0x000055a99343cd64 in ap_process_request (r=r@entry=0x55a996501960) at
http_request.c:363
#17 0x000055a993438f62 in ap_process_http_sync_connection
(c=0x55a9964a9090) at http_core.c:190
#18 ap_process_http_connection (c=0x55a9964a9090) at http_core.c:231
#19 0x000055a993430fc0 in ap_run_process_connection (c=c@entry=0x55a9964a9090)
at connection.c:41
#20 0x000055a9934313a8 in ap_process_connection (c=c@entry=0x55a9964a9090,
csd=<optimized out>) at connection.c:202
#21 0x00007f24769547af in child_main (child_num_arg=child_num_arg@entry=6)
at prefork.c:707
#22 0x00007f24769549f5 in make_child (s=0x55a994af9e30, slot=6) at
prefork.c:810
#23 0x00007f247695568e in perform_idle_server_maintenance (p=<optimized
out>) at prefork.c:912
#24 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized
out>) at prefork.c:1100
#25 0x000055a99340bffe in ap_run_mpm (pconf=pconf@entry=0x55a994ab5138,
plog=0x55a994ae2358, s=0x55a994af9e30) at mpm_common.c:96
#26 0x000055a993404d76 in main (argc=2, argv=0x7ffd5cd97878) at main.c:783
*bt full for 10 and 11:*
#10 0x00007f2474eef7f9 in proxy_http_handler (r=<optimized out>,
worker=<optimized out>, conf=0x55a994beaef0, url=0x55a9964d819e "
http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming
",
proxyname=0x0, proxyport=0) at mod_proxy_http.c:2038
locurl = 0x55a9964d8490
"/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming"
status = <optimized out>
server_portstr =
"\000vM\226\251U\000\000\350\030P\226\251U\000\000\000q\331\\\375\177\000\000\000\000\000\000\000\000\000"
scheme = <optimized out>
proxy_function = 0x7f2474ef0bfa "HTTP"
u = <optimized out>
backend = 0x55a9964b5600
is_ssl = 0
c = 0x55a9964a9090
retry = 0
p = <optimized out>
uri = 0x55a9964d83b0
#11 0x00007f247673edc4 in proxy_run_scheme_handler (r=r@entry=0x55a996501960,
worker=0x55a994beb1b0, conf=conf@entry=0x55a994beaef0,
url=0x55a9964d819e "
http://nike.pbtech:3838/aln2tree/__sockjs__/n=zlsc268kyOEGKXGAXe/863/48gpkzl4/xhr_streaming";,
proxyhost=proxyhost@entry=0x0, proxyport=proxyport@entry=0) at
mod_proxy.c:2746
pHook = <optimized out>
n = 3
rv = -1
I am beyond my knowledge limit at this point.
The server nike.pbtech contains an R application running on internal
server. It's reached through this Apace proxy.
I have a few questions.
1) How do we tell the apache fork's hung hung?
2) How can I check all forks at once? Attaching to parent does not work.
3) When trying other Apache forks I get hung gdb at line "attaching to
process." Any way to avoid?
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit
Physiology and Biophysics
Weill Cornell Medicine
E: d...@med.cornell.edu
O: 212-746-6305
F: 212-746-8690
On Mon, Nov 13, 2017 at 7:04 AM, Rainer Canavan <rainer.cana...@sevenval.com
> wrote:
> On Fri, Nov 10, 2017 at 6:41 PM, Douglas Duckworth
> <dod2...@med.cornell.edu> wrote:
> > Hi
> >
> > I am running old PHP under Apache httpd-2.4.
>
> [...]
> > Though, ever few weeks, we see sudden increase in workers who never seem
> to
> > retire:
> >
> > [Fri Nov 10 02:43:20.019924 2017] [mpm_prefork:error] [pid 13584]
> AH00161:
> > server reached MaxRequestWorkers setting, consider raising the
> > MaxRequestWorkers setting
> >
> > user@server[/var/www]$ ps aux | grep [h]ttpd | wc -l
> > 257
>
> If the php locks up while processing your request, no logs will be
> written. You may be running
> into a bug where circular, unresolvable dependencies for a lock
> prevent the processes from
> completing their requests. To check what's going on, install gdb, the
> debug info for your php
> and httpd and find the .gdbinfo that came with the httpd and php
> version you're using. Then
> attach gdb to any of the hanging processes (gdb `which httpd` PID),
> source both .gdbinit files,
> do a "zbacktrace" and a "bt full", and repeat for some other hanging
> processes. Depending
> on the type of lock, you may be able to identify the first process
> that has acquired that lock
> that all others are waiting for, and the php code and / or php module
> that causes it.
>
> rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
> For additional commands, e-mail: users-h...@httpd.apache.org
>
>
