Ok, I got a stack trace. My understanding of this is limited, but the thread seems to be stuck at polling something without timeout? (gdb reports that file ../sysdeps/unix/sysv/linux/poll.c cannot be found)
The full session follows. (Using plaintext email today in the hope that this helps with readibility. Please let me know if I should send differently instead.) Between the two "thread apply all bt" commands I waited about two minutes: $ sudo gdb /usr/sbin/apache2 60501 GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1 Copyright (C) 2022 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/apache2... (No debugging symbols found in /usr/sbin/apache2) Attaching to program: /usr/sbin/apache2, process 60501 [New LWP 60532] [New LWP 60533] [New LWP 60534] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 29 ../sysdeps/unix/sysv/linux/poll.c: Datei oder Verzeichnis nicht gefunden. (gdb) thread apply all bt Thread 4 (Thread 0x7f68ad6f9640 (LWP 60534) "apache2"): #0 __futex_abstimed_wait_common64 (private=-1359973880, cancel=true, abstime=0x7f68ad6f7220, op=137, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-1359973880, abstime=0x7f68ad6f7220, clockid=-1359689157, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f68ad6f7220, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68ad6f7220, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68ad6f7220) at ./nptl/pthread_cond_wait.c:652 #5 0x00007f68aefe5df5 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #6 0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #7 0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #8 0x00007f68ac30d6e3 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so #9 0x00007f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0 #10 0x00007f68a750b2d6 in ?? () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0 #11 0x00007f68ac3087d6 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so #12 0x00007f68aef118ba in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #13 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #14 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #15 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #16 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #17 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #18 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #19 0x00007f68aee9c9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #20 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #21 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #22 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #23 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #24 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #25 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #26 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #27 0x00007f68aef08118 in PyVectorcall_Call () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #28 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #29 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #30 0x00007f68aee9962e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #31 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #32 0x00007f68aef08618 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #33 0x00007f68aee99458 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #34 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #35 0x00007f68aef086ac in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #36 0x00007f68af41680f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #37 0x00007f68af41c9c8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #38 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #39 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Thread 3 (Thread 0x7f68adefa640 (LWP 60533) "apache2"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f68adef9d80, op=137, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7f68adef9d80, clockid=0, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f68adef9d80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68adef9d80, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68adef9d80) at ./nptl/pthread_cond_wait.c:652 #5 0x00007f68aefe5df5 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #6 0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #7 0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #8 0x00007f68af41cf30 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #9 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #10 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Thread 2 (Thread 0x7f68ae6fb640 (LWP 60532) "apache2"): #0 0x00007f68af72e7ed in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7f68ae6fad60) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x00007f68af865c09 in apr_sleep () from target:/lib/x86_64-linux-gnu/libapr-1.so.0 --Type <RET> for more, q to quit, c to continue without paging-- #2 0x00007f68af41d11c in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #3 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #4 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Thread 1 (Thread 0x7f68af59a780 (LWP 60501) "apache2"): #0 0x00007f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f68af85e0fb in apr_poll () from target:/lib/x86_64-linux-gnu/libapr-1.so.0 #2 0x00007f68af4222d8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #3 0x00007f68af423f1f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #4 0x000055c9771cc73d in ap_run_pre_mpm () #5 0x00007f68af478455 in ?? () from target:/usr/lib/apache2/modules/mod_mpm_event.so #6 0x000055c9771c30e8 in ap_run_mpm () #7 0x000055c9771c2609 in main () (gdb) (gdb) (gdb) (gdb) (gdb) thread apply all bt Thread 4 (Thread 0x7f68ad6f9640 (LWP 60534) "apache2"): #0 __futex_abstimed_wait_common64 (private=-1359973880, cancel=true, abstime=0x7f68ad6f7220, op=137, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=-1359973880, abstime=0x7f68ad6f7220, clockid=-1359689157, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f68ad6f7220, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68ad6f7220, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68ad6f7220) at ./nptl/pthread_cond_wait.c:652 #5 0x00007f68aefe5df5 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #6 0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #7 0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #8 0x00007f68ac30d6e3 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so #9 0x00007f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0 #10 0x00007f68a750b2d6 in ?? () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0 #11 0x00007f68ac3087d6 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so #12 0x00007f68aef118ba in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #13 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #14 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #15 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #16 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #17 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #18 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #19 0x00007f68aee9c9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #20 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #21 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #22 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #23 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #24 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #25 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #26 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #27 0x00007f68aef08118 in PyVectorcall_Call () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #28 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #29 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #30 0x00007f68aee9962e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #31 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #32 0x00007f68aef08618 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #33 0x00007f68aee99458 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #34 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #35 0x00007f68aef086ac in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #36 0x00007f68af41680f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #37 0x00007f68af41c9c8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #38 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #39 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Thread 3 (Thread 0x7f68adefa640 (LWP 60533) "apache2"): #0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f68adef9d80, op=137, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7f68adef9d80, clockid=0, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f68adef9d80, private=private@entry=0) at ./nptl/futex-internal.c:139 #3 0x00007f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68adef9d80, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503 #4 ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68adef9d80) at ./nptl/pthread_cond_wait.c:652 #5 0x00007f68aefe5df5 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #6 0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #7 0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0 #8 0x00007f68af41cf30 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #9 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #10 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Thread 2 (Thread 0x7f68ae6fb640 (LWP 60532) "apache2"): #0 0x00007f68af72e7ed in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7f68ae6fad60) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x00007f68af865c09 in apr_sleep () from target:/lib/x86_64-linux-gnu/libapr-1.so.0 --Type <RET> for more, q to quit, c to continue without paging-- #2 0x00007f68af41d11c in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #3 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442 #4 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 Thread 1 (Thread 0x7f68af59a780 (LWP 60501) "apache2"): #0 0x00007f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f68af85e0fb in apr_poll () from target:/lib/x86_64-linux-gnu/libapr-1.so.0 #2 0x00007f68af4222d8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #3 0x00007f68af423f1f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so #4 0x000055c9771cc73d in ap_run_pre_mpm () #5 0x00007f68af478455 in ?? () from target:/usr/lib/apache2/modules/mod_mpm_event.so #6 0x000055c9771c30e8 in ap_run_mpm () #7 0x000055c9771c2609 in main () (gdb) Best regards, Carsten Am 07.02.23 um 22:44 schrieb Graham Dumpleton: > Only other thing can suggest if you know point at which it is hanging, is to > test and extract C stack trace by attaching gdb to the running process. This > may or may not work depending on your system setup and what tools are > installed. > > Debugging Techniques — mod_wsgi 4.9.4 documentation > <https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb> > modwsgi.readthedocs.io > <https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb> > favicon.ico > <https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb> > > <https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb> > > > That may give some clues. > > Graham > >> On 8 Feb 2023, at 8:42 am, Graham Dumpleton <graham.dumple...@gmail.com> >> wrote: >> >> Is any of your code using ctypes module for Apache, or any other module >> which is used to call into C code by writing Python code only? >> >> Graham >> -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/a09f8380-eddd-e859-911b-5919d44ea610%40cafu.de.
