https://bz.apache.org/bugzilla/show_bug.cgi?id=65180
Bug ID: 65180
Summary: mod_proxy_wstunnel stuck processes at graceful
stopping
Product: Apache httpd-2
Version: 2.4.25
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_proxy_wstunnel
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
Created attachment 37768
--> https://bz.apache.org/bugzilla/attachment.cgi?id=37768&action=edit
Apache status summary
Hi,
I noticed that our production server has several processes in the "stop: yes
(old gen)" state, which leads to excessive memory consumption.
Here is the scoreboard under normal load (screenshot attached):
G..GGG.G.G.G.GG..G.G..G.G........G.G.GGGGG..G.GG.GG.GG...GG.....
..........................G.............................WWW_WWW_
_WWWW__W__W______WWW__W___________WWW_W___W_W_W____W___W_R_____R
___W____________W____________W__________WR_____W..GG.G..GGG.....
.G............G...G....GG.G.G...................................
............G.........G....................GG...G..G.........G..
G.G.......G.......GG.G....GG....................................
.........................G......................................
..............................................................G.
................................................................
................................................................
................................................................
................................................................
................................................................
....
I've investigated the issue with gdb, and it appears that some websocket
connections handled by proxy_wstunnel are not properly released (trace attached
below).
Details about our configuration :
Server version: Apache/2.4.25 (Debian)
Server built: 2019-10-13T15:43:54
Architecture: 64-bit
Server MPM: event
threaded: yes (fixed thread count)
forked: yes (variable process count)
- event MPM config
<IfModule mpm_event_module>
ServerLimit 16
ThreadLimit 64
ThreadsPerChild 60
StartServers 2
MaxRequestWorkers 900
MinSpareThreads 30
MaxSpareThreads 90
AsyncRequestWorkerFactor 3
MaxConnectionsPerChild 10000
</IfModule>
- mod_proxy_wstunnel config
<VirtualHost _default_:443>
# ...
ProxyPass /socket.io/ ws://127.0.0.1:5555/socket.io/
ProxyPassReverse /socket.io/ ws://127.0.0.1:5555/socket.io/
</VirtualHost>
- GDB `thread apply all bt` of one PID
Thread 4 (Thread 0x7f44887c8700 (LWP 32758)):
#0 0x00007f44b1642303 in epoll_wait ()
at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f44b1b38201 in ?? ()
from target:/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f44ae860fdc in ?? ()
from target:/usr/lib/apache2/modules/mod_mpm_event.so
#3 0x00007f44b18ff4a4 in start_thread (arg=0x7f44887c8700)
at pthread_create.c:456
#4 0x00007f44b1641d0f in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 3 (Thread 0x7f44907d8700 (LWP 32742)):
#0 0x00007f44b1642303 in epoll_wait ()
at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f44b1b38201 in ?? ()
from target:/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f44ae024d83 in ?? ()
from target:/usr/lib/apache2/modules/mod_proxy_wstunnel.so
#3 0x00007f44ae43c2bc in proxy_run_scheme_handler ()
from target:/usr/lib/apache2/modules/mod_proxy.so
#4 0x00007f44ae43d251 in ?? ()
---Type <return> to continue, or q <return> to quit---
o
#5 0x000055f06836dd60 in ap_run_handler ()
#6 0x000055f06836e2f6 in ap_invoke_handler ()
#7 0x000055f068385db3 in ap_process_async_request ()
#8 0x000055f068382209 in ?? ()
#9 0x000055f068377bd0 in ap_run_process_connection ()
#10 0x00007f44ae85e6e8 in ?? ()
from target:/usr/lib/apache2/modules/mod_mpm_event.so
#11 0x00007f44b18ff4a4 in start_thread (arg=0x7f44907d8700)
at pthread_create.c:456
#12 0x00007f44b1641d0f in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 2 (Thread 0x7f44a9253700 (LWP 32701)):
#0 0x00007f44b1642303 in epoll_wait ()
at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f44b1b38201 in ?? ()
from target:/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f44ae024d83 in ?? ()
from target:/usr/lib/apache2/modules/mod_proxy_wstunnel.so
#3 0x00007f44ae43c2bc in proxy_run_scheme_handler ()
from target:/usr/lib/apache2/modules/mod_proxy.so
#4 0x00007f44ae43d251 in ?? ()
from target:/usr/lib/apache2/modules/mod_proxy.so
#5 0x000055f06836dd60 in ap_run_handler ()
#6 0x000055f06836e2f6 in ap_invoke_handler ()
#7 0x000055f068385db3 in ap_process_async_request ()
#8 0x000055f068382209 in ?? ()
#9 0x000055f068377bd0 in ap_run_process_connection ()
#10 0x00007f44ae85e6e8 in ?? ()
from target:/usr/lib/apache2/modules/mod_mpm_event.so
#11 0x00007f44b18ff4a4 in start_thread (arg=0x7f44a9253700)
at pthread_create.c:456
#12 0x00007f44b1641d0f in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 1 (Thread 0x7f44b23f6040 (LWP 32508)):
#0 0x00007f44b19006dd in pthread_join (threadid=139932324366080,
thread_return=0x7ffe0416bb00) at pthread_join.c:90
---Type <return> to continue, or q <return> to quit---
#1 0x00007f44b1b3cb5b in apr_thread_join ()
from target:/usr/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f44ae85c247 in ?? ()
from target:/usr/lib/apache2/modules/mod_mpm_event.so
#3 0x00007f44ae85c67e in ?? ()
from target:/usr/lib/apache2/modules/mod_mpm_event.so
#4 0x00007f44ae85fc99 in ?? ()
from target:/usr/lib/apache2/modules/mod_mpm_event.so
#5 0x00007f44ae85fd3c in ?? ()
from target:/usr/lib/apache2/modules/mod_mpm_event.so
#6 0x00007f44ae860a6b in ?? ()
from target:/usr/lib/apache2/modules/mod_mpm_event.so
#7 0x000055f0683510fe in ap_run_mpm ()
#8 0x000055f068349cfd in main ()
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
