Can you try and get a full backtrace for all threads using "thread apply all bt" as described in:
* https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb Really need the context with all stack frames so can narrow down which package the problem occurs in. Graham > On 17 Oct 2024, at 6:37 AM, Molly Foley <[email protected]> wrote: > > Hi Graham, thanks for the reply. We do not have any PHP modules enabled on > our Apache server, so I don't think that's it. > > That's interesting to hear about the Anaconda distribution and I can see how > that would be true for the "official" distribution, but is that also true for > community driven distributions like miniforge3? They only use the conda-forge > channel for installing packages in all environments, including the base > environment, unless you specifically request it install from some other > channel (we've never needed to). If miniforge3 doesn't work, I might try > micromamba > <https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html>instead > and see if that makes a difference somehow. We don't like using venv because > many scientific packages are an absolute pain to install using pip, such as > gdal. > On Tuesday, October 15, 2024 at 4:22:40 PM UTC-4 Graham Dumpleton wrote: >> Do you have PHP enabled for the same Apache? >> >> Generally I discourage use of Anaconda Python distributions as they do >> things which break embedding for Python. >> >> One of the problems is that Anaconda Python ships its own versions of some >> libraries which are usually installed system wide. The problem is that PHP >> loads the system wide variants and when the Anaconda versions are >> incompatible, the process trying to use the Anaconda version will crash. >> >> So if you have PHP enabled, are you able to disable it so it isn't loaded by >> Apache? >> >> Not that it will help, but since you are using mod_wsgi daemon mode, also >> add outside of VirtualHost definitions (good spot is after the LoadModule), >> the directive: >> >> WSGIRestrictEmbedded On >> >> That will mean Python isn't initialised in Apache child worker processes >> where it isn't needed. >> >> Graham >> >> >>> On 16 Oct 2024, at 6:30 AM, Molly Foley <[email protected] <>> wrote: >>> >> >>> I'm struggling to get multiple conda environments working and I'm hoping to >>> get some insight here. >>> >>> I have installed python with the miniforge3 distribution (conda, >>> basically). In the base environment, I have installed mod_wsgi 5.0.0 which >>> appears to compile again python 3.12. The base python environment has >>> python 3.12.6 installed (I left it as default because messing with the base >>> environment is usually a no-no). Additionally, I have created two conda >>> virtual environments, one for each virtual host I have in Apache. Each >>> virtual environment has python 3.12.7 installed. >>> >>> sys.prefix is /opt/miniforge3 >>> >>> As root user, echo $PATH results in: >>> /root/.nvm/versions/node/v20.5.0/bin:/root/.local/bin:/root/bin:/opt/miniforge3/bin/:/sbin:/bin:/usr/sbin:/usr/bin:/var/cfengine/bin >>> >>> >>> >>> LoadModule config looks like this: >>> >>> # NOTE: mod_wsgi_python3 can not coexist in the same apache process as >>> >>> # mod_wsgi (python2). Only load if mod_wsgi is not already loaded. >>> >>> <IfModule !wsgi_module> >>> >>> LoadModule wsgi_module >>> /opt/miniforge3/lib/python3.12/site-packages/mod_wsgi/server/mod_wsgi-py312.cpython-312-x86_64-linux-gnu.so >>> <http://mod_wsgi-py312.cpython-312-x86_64-linux-gnu.so/> >>> >>> </IfModule> >>> >>> >>> >>> In Apache, I have set the apache LogLevel to info. I have also set the >>> virtual hosts' specific logs to LogLevel to info. >>> >>> Here is the relevant config for the virtual hosts: >>> <VirtualHost *:443> >>> >>> ServerName myServerName.com <http://myservername.com/> >>> ## Vhost docroot >>> >>> DocumentRoot "/var/www/vhosts/hydrosource_chap" >>> >>> ## Logging >>> >>> ErrorLog "/var/log/httpd/chap-error.log" >>> >>> LogLevel info >>> >>> ServerSignature Off >>> >>> CustomLog "/var/log/httpd/chap-access.log" combined >>> >>> ## WSGI configuration >>> >>> WSGIApplicationGroup %{GLOBAL} >>> >>> WSGIDaemonProcess hydrosource_chap display-name=%{GROUP} >>> home=/var/www/vhosts/hydrosource_chap python-home=/opt/miniforge3/envs/chap >>> threads=1 user=hydro >>> >>> WSGIProcessGroup hydrosource_chap >>> >>> WSGIScriptAlias / "/var/www/vhosts/hydrosource_chap/wsgi.py" >>> >>> ## Anything start with dot >>> >>> <DirectoryMatch "^\.|\/\."> >>> >>> Require all denied >>> >>> </DirectoryMatch> >>> >>> >>> >>> <LocationMatch "\/\."> >>> >>> Require all denied >>> >>> </LocationMatch> >>> >>> >>> >>> <Location /> >>> >>> Require all granted >>> >>> </Location> >>> >>> </VirtualHost> >>> >>> <VirtualHost *:443> >>> >>> ServerName mySecondName.com <http://mysecondname.com/> >>> ## Vhost docroot >>> >>> DocumentRoot "/var/www/vhosts/hydrosource-2" >>> >>> ## Logging >>> >>> ErrorLog "/var/log/httpd/dataexplorer-error.log" >>> >>> LogLevel info >>> >>> ServerSignature Off >>> >>> CustomLog "/var/log/httpd/dataexplorer-access.log" combined >>> >>> ## WSGI configuration >>> >>> WSGIApplicationGroup %{GLOBAL} >>> >>> WSGIDaemonProcess dataexplorer display-name=%{GROUP} >>> home=/var/www/vhosts/hydrosource-2 >>> python-home=/opt/miniforge3/envs/dataexplorer threads=1 user=hydro >>> >>> WSGIProcessGroup dataexplorer >>> >>> WSGIScriptAlias / "/var/www/vhosts/hydrosource-2/wsgi.py" >>> >>> ## Anything start with dot >>> >>> <DirectoryMatch "^\.|\/\."> >>> >>> Require all denied >>> >>> </DirectoryMatch> >>> >>> >>> >>> <LocationMatch "\/\."> >>> >>> Require all denied >>> >>> </LocationMatch> >>> >>> >>> >>> <Location /> >>> >>> Require all granted >>> >>> </Location> >>> >>> </VirtualHost> >>> >>> >>> >>> When I systemctl restart httpd and try to load the dataexplorer >>> application, for instance, I get segmentation fault errors in the apache >>> error_log and issues with oversized headers in the virtual host specific >>> log. >>> >>> >>> >>> Apache Log: >>> >>> [Tue Oct 15 15:10:34.242996 2024] [wsgi:info] [pid 568871:tid 568871] >>> mod_wsgi (pid=568871): Starting process 'hydrosource_chap' with uid=36143, >>> gid=48 and threads=1. >>> >>> [Tue Oct 15 15:10:34.243772 2024] [wsgi:info] [pid 568872:tid 568872] >>> mod_wsgi (pid=568872): Starting process 'dataexplorer' with uid=36143, >>> gid=48 and threads=1. >>> >>> [Tue Oct 15 15:10:34.246679 2024] [mpm_event:notice] [pid 568866:tid >>> 568866] AH00489: Apache/2.4.57 (Red Hat Enterprise Linux) OpenSSL/3.0.7 >>> mod_wsgi/5.0.0 Python/3.12 configured -- resuming normal operations >>> >>> [Tue Oct 15 15:10:34.246702 2024] [mpm_event:info] [pid 568866:tid 568866] >>> AH00490: Server built: Aug 5 2024 00:00:00 >>> >>> [Tue Oct 15 15:10:34.246719 2024] [core:notice] [pid 568866:tid 568866] >>> AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' >>> >>> [Tue Oct 15 15:10:34.248317 2024] [wsgi:info] [pid 568871:tid 568871] >>> mod_wsgi (pid=568871): Python home /opt/miniforge3/envs/chap. >>> >>> [Tue Oct 15 15:10:34.248321 2024] [wsgi:info] [pid 568872:tid 568872] >>> mod_wsgi (pid=568872): Python home /opt/miniforge3/envs/dataexplorer. >>> >>> [Tue Oct 15 15:10:34.248377 2024] [wsgi:info] [pid 568871:tid 568871] >>> mod_wsgi (pid=568871): Initializing Python. >>> >>> [Tue Oct 15 15:10:34.248387 2024] [wsgi:info] [pid 568872:tid 568872] >>> mod_wsgi (pid=568872): Initializing Python. >>> >>> [Tue Oct 15 15:10:34.249092 2024] [http2:info] [pid 568874:tid 568874] >>> h2_workers: created with min=25 max=37 idle_ms=600000 >>> >>> [Tue Oct 15 15:10:34.250137 2024] [wsgi:info] [pid 568874:tid 568874] >>> mod_wsgi (pid=568874): Initializing Python. >>> >>> [Tue Oct 15 15:10:34.256663 2024] [http2:info] [pid 568875:tid 568875] >>> h2_workers: created with min=25 max=37 idle_ms=600000 >>> >>> [Tue Oct 15 15:10:34.257760 2024] [wsgi:info] [pid 568875:tid 568875] >>> mod_wsgi (pid=568875): Initializing Python. >>> >>> [Tue Oct 15 15:10:34.258652 2024] [http2:info] [pid 568873:tid 568873] >>> h2_workers: created with min=25 max=37 idle_ms=600000 >>> >>> [Tue Oct 15 15:10:34.259732 2024] [wsgi:info] [pid 568873:tid 568873] >>> mod_wsgi (pid=568873): Initializing Python. >>> >>> [Tue Oct 15 15:10:34.262413 2024] [wsgi:info] [pid 568874:tid 568874] >>> mod_wsgi (pid=568874): Attach interpreter ''. >>> >>> [Tue Oct 15 15:10:34.271322 2024] [wsgi:info] [pid 568873:tid 568873] >>> mod_wsgi (pid=568873): Attach interpreter ''. >>> >>> [Tue Oct 15 15:10:34.277995 2024] [wsgi:info] [pid 568875:tid 568875] >>> mod_wsgi (pid=568875): Attach interpreter ''. >>> >>> [Tue Oct 15 15:10:34.279542 2024] [wsgi:info] [pid 568874:tid 568874] >>> mod_wsgi (pid=568874): Imported 'mod_wsgi'. >>> >>> [Tue Oct 15 15:10:34.288265 2024] [wsgi:info] [pid 568873:tid 568873] >>> mod_wsgi (pid=568873): Imported 'mod_wsgi'. >>> >>> [Tue Oct 15 15:10:34.296894 2024] [wsgi:info] [pid 568875:tid 568875] >>> mod_wsgi (pid=568875): Imported 'mod_wsgi'. >>> >>> [Tue Oct 15 15:10:49.266694 2024] [http2:info] [pid 569108:tid 569108] >>> h2_workers: created with min=25 max=37 idle_ms=600000 >>> >>> [Tue Oct 15 15:10:49.267916 2024] [wsgi:info] [pid 569108:tid 569108] >>> mod_wsgi (pid=569108): Initializing Python. >>> >>> [Tue Oct 15 15:10:49.281851 2024] [wsgi:info] [pid 569108:tid 569108] >>> mod_wsgi (pid=569108): Attach interpreter ''. >>> >>> [Tue Oct 15 15:10:49.298865 2024] [wsgi:info] [pid 569108:tid 569108] >>> mod_wsgi (pid=569108): Imported 'mod_wsgi'. >>> >>> [Tue Oct 15 15:10:50.264356 2024] [core:notice] [pid 568866:tid 568866] >>> AH00051: child pid 568872 exit signal Segmentation fault (11), possible >>> coredump in /etc/httpd >>> >>> [Tue Oct 15 15:10:50.264458 2024] [wsgi:info] [pid 568866:tid 568866] >>> mod_wsgi (pid=568872): Process 'dataexplorer' has died, deregister and >>> restart it. >>> >>> [Tue Oct 15 15:10:50.264466 2024] [wsgi:info] [pid 568866:tid 568866] >>> mod_wsgi (pid=568872): Process 'dataexplorer' terminated by signal 11 >>> >>> [Tue Oct 15 15:10:50.264473 2024] [wsgi:info] [pid 568866:tid 568866] >>> mod_wsgi (pid=568872): Process 'dataexplorer' has been deregistered and >>> will no longer be monitored. >>> >>> [Tue Oct 15 15:10:50.265514 2024] [wsgi:info] [pid 569173:tid 569173] >>> mod_wsgi (pid=569173): Starting process 'dataexplorer' with uid=36143, >>> gid=48 and threads=1. >>> >>> [Tue Oct 15 15:10:50.268450 2024] [wsgi:info] [pid 569173:tid 569173] >>> mod_wsgi (pid=569173): Python home /opt/miniforge3/envs/dataexplorer. >>> >>> [Tue Oct 15 15:10:50.268513 2024] [wsgi:info] [pid 569173:tid 569173] >>> mod_wsgi (pid=569173): Initializing Python. >>> >>> [Tue Oct 15 15:10:51.265981 2024] [core:notice] [pid 568866:tid 568866] >>> AH00051: child pid 569173 exit signal Segmentation fault (11), possible >>> coredump in /etc/httpd >>> >>> [Tue Oct 15 15:10:51.266037 2024] [wsgi:info] [pid 568866:tid 568866] >>> mod_wsgi (pid=569173): Process 'dataexplorer' has died, deregister and >>> restart it. >>> >>> [Tue Oct 15 15:10:51.266044 2024] [wsgi:info] [pid 568866:tid 568866] >>> mod_wsgi (pid=569173): Process 'dataexplorer' terminated by signal 11 >>> >>> [Tue Oct 15 15:10:51.266050 2024] [wsgi:info] [pid 568866:tid 568866] >>> mod_wsgi (pid=569173): Process 'dataexplorer' has been deregistered and >>> will no longer be monitored. >>> >>> [Tue Oct 15 15:10:51.267079 2024] [wsgi:info] [pid 569188:tid 569188] >>> mod_wsgi (pid=569188): Starting process 'dataexplorer' with uid=36143, >>> gid=48 and threads=1. >>> >>> [Tue Oct 15 15:10:51.269694 2024] [wsgi:info] [pid 569188:tid 569188] >>> mod_wsgi (pid=569188): Python home /opt/miniforge3/envs/dataexplorer. >>> >>> >>> [Tue Oct 15 15:10:51.269761 2024] [wsgi:info] [pid 569188:tid 569188] >>> mod_wsgi (pid=569188): Initializing Python. >>> >>> >>> >>> dataexplorer log: >>> >>> [Tue Oct 15 15:10:34.267712 2024] [wsgi:info] [pid 568872:tid 568872] >>> mod_wsgi (pid=568872): Attach interpreter ''. >>> >>> [Tue Oct 15 15:10:48.949190 2024] [wsgi:info] [pid 568872:tid 568981] >>> [remote 10.159.64.61:59510 <http://10.159.64.61:59510/>] mod_wsgi >>> (pid=568872, process='dataexplorer', application=''): Loading Python script >>> file '/var/www/vhosts/hydrosource-2/wsgi.py'. >>> >>> [Tue Oct 15 15:10:49.636199 2024] [wsgi:error] [pid 568875:tid 569031] >>> [client 10.159.64.61:59510 <http://10.159.64.61:59510/>] Truncated or >>> oversized response headers received from daemon process 'dataexplorer': >>> /var/www/vhosts/hydrosource-2/wsgi.py >>> >>> [Tue Oct 15 15:10:50.282619 2024] [wsgi:info] [pid 569173:tid 569173] >>> mod_wsgi (pid=569173): Attach interpreter ''. >>> >>> [Tue Oct 15 15:10:50.293954 2024] [wsgi:info] [pid 569173:tid 569176] >>> [remote 10.159.64.61:59511 <http://10.159.64.61:59511/>] mod_wsgi >>> (pid=569173, process='dataexplorer', application=''): Loading Python script >>> file '/var/www/vhosts/hydrosource-2/wsgi.py'. >>> >>> [Tue Oct 15 15:10:50.956465 2024] [wsgi:error] [pid 568874:tid 568980] >>> [client 10.159.64.61:59511 <http://10.159.64.61:59511/>] Truncated or >>> oversized response headers received from daemon process 'dataexplorer': >>> /var/www/vhosts/hydrosource-2/wsgi.py, referer: >>> https://dataexplorer.hydroclimate-dev.ornl.gov/ >>> >>> >>> [Tue Oct 15 15:10:51.281032 2024] [wsgi:info] [pid 569188:tid 569188] >>> mod_wsgi (pid=569188): Attach interpreter ''. >>> >>> >>> >>> Given the segmentation faults, I do a core dump and then use gdb to look at >>> the problem. To be honest, I don't really know what I'm looking at here. If >>> I do "bt", it prints out a lot more stuff, but I can't make heads or tails >>> of it. This is the output without backtracing with "bt". >>> >>> GNU gdb (GDB) Red Hat Enterprise Linux 10.2-13.el9 >>> >>> Copyright (C) 2021 Free Software Foundation, Inc. >>> >>> License GPLv3+: GNU GPL version 3 or later >>> <http://gnu.org/licenses/gpl.html> >>> >>> This is free software: you are free to change and redistribute it. >>> >>> There is NO WARRANTY, to the extent permitted by law. >>> >>> Type "show copying" and "show warranty" for details. >>> >>> This GDB was configured as "x86_64-redhat-linux-gnu". >>> >>> Type "show configuration" for configuration details. >>> >>> For bug reporting instructions, please see: >>> >>> <https://www.gnu.org/software/gdb/bugs/>. >>> >>> Find the GDB manual and other documentation resources online at: >>> >>> <http://www.gnu.org/software/gdb/documentation/>. >>> >>> >>> >>> For help, type "help". >>> >>> Type "apropos word" to search for commands related to "word"... >>> >>> Reading symbols from /usr/sbin/httpd... >>> >>> Reading symbols from .gnu_debugdata for /usr/sbin/httpd... >>> >>> (No debugging symbols found in .gnu_debugdata for /usr/sbin/httpd) >>> >>> >>> >>> warning: Can't open file /dev/zero (deleted) during file-backed mapping >>> note processing >>> >>> [New LWP 568981] >>> >>> [New LWP 568872] >>> >>> [New LWP 568979] >>> >>> [New LWP 568977] >>> >>> [Thread debugging using libthread_db enabled] >>> >>> Using host libthread_db library "/lib64/libthread_db.so >>> <http://libthread_db.so/>.1". >>> >>> Core was generated by `(wsgi:dataexplo -DFOREGROUND'. >>> >>> Program terminated with signal SIGSEGV, Segmentation fault. >>> >>> #0 PyObject_GetAttr (v=0x7fa2e8a44130, name=0x7fa2eae33c20 >>> <_PyRuntime+45280>) at >>> /usr/local/src/conda/python-3.12.6/Include/object.h:968 >>> >>> 968 /usr/local/src/conda/python-3.12.6/Include/object.h: No such file or >>> directory. >>> >>> [Current thread is 1 (Thread 0x7fa2e9425640 (LWP 568981))] >>> >>> >>> Missing separate debuginfos, use: dnf debuginfo-install >>> httpd-core-2.4.57-11.el9_4.1.x86_64 >>> >>> >>> >>> It seems like it's routing to an incorrect python install somehow. >>> /usr/local/src/conda is not a path that exists on this machine, so the >>> error makes sense, but I don't know what it's trying to do on this line. >>> >>> I have attempted to set header-buffer-size on my WSGIProcessDaemon >>> directives but haven't had any luck with that. If I remove the offending >>> packages (things that have to do with SQL like flask-sqlalchemy, psycopg2, >>> geoalchemy2, etc) then the apps will run without crashing, but will >>> obviously give me errors about being unable to import the packages I >>> removed since they're needed in my app. I'm really not sure how to move >>> forward on this. Is there something obviously wrong in my configuration? Do >>> I need to install mod_wsgi in each virtual environment instead of the base >>> environment (and if so, then how would LoadModule work in Apache?). >>> >>> >>> >>> Lastly, here's my wsgi.py script file (each app has the same one). I am >>> using blueprints in my Flask app, so I import the create_app function to >>> generate the application. >>> >>> import sys >>> from api import create_app >>> from os import getcwd >>> >>> sys.path.insert(0, getcwd() + "/api") >>> sys.path.append(getcwd() + "/api/models") >>> >>> application = create_app() >>> >>> Any insight on how to debug this would be very helpful. I have reviewed the >>> debugging technique docs, but I really feel lost with the gnu debugger. >>> >>> Thanks for your time! >>> >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] <>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/b6af7e32-21b7-4f04-9e5b-ab1e8cfaa7dan%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/b6af7e32-21b7-4f04-9e5b-ab1e8cfaa7dan%40googlegroups.com?utm_medium=email&utm_source=footer>. >> > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/a96269ef-2e4a-4c4c-9a1f-877cf9e65e04n%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/a96269ef-2e4a-4c4c-9a1f-877cf9e65e04n%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/1C771164-0DE0-40AB-A12F-2AB80C7DCC88%40gmail.com.
