Hi Graham, thanks for the reply. We do not have any PHP modules enabled on our Apache server, so I don't think that's it.
That's interesting to hear about the Anaconda distribution and I can see how that would be true for the "official" distribution, but is that also true for community driven distributions like miniforge3? They only use the conda-forge channel for installing packages in all environments, including the base environment, unless you specifically request it install from some other channel (we've never needed to). If miniforge3 doesn't work, I might try micromamba <https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html> instead and see if that makes a difference somehow. We don't like using venv because many scientific packages are an absolute pain to install using pip, such as gdal. On Tuesday, October 15, 2024 at 4:22:40 PM UTC-4 Graham Dumpleton wrote: > Do you have PHP enabled for the same Apache? > > Generally I discourage use of Anaconda Python distributions as they do > things which break embedding for Python. > > One of the problems is that Anaconda Python ships its own versions of some > libraries which are usually installed system wide. The problem is that PHP > loads the system wide variants and when the Anaconda versions are > incompatible, the process trying to use the Anaconda version will crash. > > So if you have PHP enabled, are you able to disable it so it isn't loaded > by Apache? > > Not that it will help, but since you are using mod_wsgi daemon mode, also > add outside of VirtualHost definitions (good spot is after the LoadModule), > the directive: > > WSGIRestrictEmbedded On > > That will mean Python isn't initialised in Apache child worker processes > where it isn't needed. > > Graham > > On 16 Oct 2024, at 6:30 AM, Molly Foley <[email protected]> wrote: > > I'm struggling to get multiple conda environments working and I'm hoping > to get some insight here. > > I have installed python with the miniforge3 distribution (conda, > basically). In the *base* environment, I have installed mod_wsgi 5.0.0 > which appears to compile again python 3.12. The *base *python environment > has python 3.12.6 installed (I left it as default because messing with the > base environment is usually a no-no). Additionally, I have created two > conda virtual environments, one for each virtual host I have in Apache. > Each virtual environment has python 3.12.7 installed. > > sys.prefix is /opt/miniforge3 > > As root user, echo $PATH results in: > > > /root/.nvm/versions/node/v20.5.0/bin:/root/.local/bin:/root/bin:/opt/miniforge3/bin/:/sbin:/bin:/usr/sbin:/usr/bin:/var/cfengine/bin > > > LoadModule config looks like this: > > # NOTE: mod_wsgi_python3 can not coexist in the same apache process as > > # mod_wsgi (python2). Only load if mod_wsgi is not already loaded. > > <IfModule !wsgi_module> > > LoadModule wsgi_module > /opt/miniforge3/lib/python3.12/site-packages/mod_wsgi/server/ > mod_wsgi-py312.cpython-312-x86_64-linux-gnu.so > > </IfModule> > > > In Apache, I have set the apache LogLevel to info. I have also set the > virtual hosts' specific logs to LogLevel to info. > > Here is the relevant config for the virtual hosts: > > <VirtualHost *:443> > > ServerName myServerName.com > > ## Vhost docroot > > DocumentRoot "/var/www/vhosts/hydrosource_chap" > > ## Logging > > ErrorLog "/var/log/httpd/chap-error.log" > > LogLevel info > > ServerSignature Off > > CustomLog "/var/log/httpd/chap-access.log" combined > > ## WSGI configuration > > WSGIApplicationGroup %{GLOBAL} > > WSGIDaemonProcess hydrosource_chap display-name=%{GROUP} > home=/var/www/vhosts/hydrosource_chap python-home=/opt/miniforge3/envs/chap > threads=1 user=hydro > > WSGIProcessGroup hydrosource_chap > > WSGIScriptAlias / "/var/www/vhosts/hydrosource_chap/wsgi.py" > > ## Anything start with dot > > <DirectoryMatch "^\.|\/\."> > > Require all denied > > </DirectoryMatch> > > > <LocationMatch "\/\."> > > Require all denied > > </LocationMatch> > > > <Location /> > > Require all granted > > </Location> > > </VirtualHost> > > <VirtualHost *:443> > > ServerName mySecondName.com > > ## Vhost docroot > > DocumentRoot "/var/www/vhosts/hydrosource-2" > > ## Logging > > ErrorLog "/var/log/httpd/dataexplorer-error.log" > > LogLevel info > > ServerSignature Off > > CustomLog "/var/log/httpd/dataexplorer-access.log" combined > > ## WSGI configuration > > WSGIApplicationGroup %{GLOBAL} > > WSGIDaemonProcess dataexplorer display-name=%{GROUP} > home=/var/www/vhosts/hydrosource-2 > python-home=/opt/miniforge3/envs/dataexplorer threads=1 user=hydro > > WSGIProcessGroup dataexplorer > > WSGIScriptAlias / "/var/www/vhosts/hydrosource-2/wsgi.py" > > ## Anything start with dot > > <DirectoryMatch "^\.|\/\."> > > Require all denied > > </DirectoryMatch> > > > <LocationMatch "\/\."> > > Require all denied > > </LocationMatch> > > > <Location /> > > Require all granted > > </Location> > > </VirtualHost> > > > When I systemctl restart httpd and try to load the dataexplorer > application, for instance, I get segmentation fault errors in the apache > error_log and issues with oversized headers in the virtual host specific > log. > > > Apache Log: > > [Tue Oct 15 15:10:34.242996 2024] [wsgi:info] [pid 568871:tid 568871] > mod_wsgi (pid=568871): Starting process 'hydrosource_chap' with uid=36143, > gid=48 and threads=1. > > [Tue Oct 15 15:10:34.243772 2024] [wsgi:info] [pid 568872:tid 568872] > mod_wsgi (pid=568872): Starting process 'dataexplorer' with uid=36143, > gid=48 and threads=1. > > [Tue Oct 15 15:10:34.246679 2024] [mpm_event:notice] [pid 568866:tid > 568866] AH00489: Apache/2.4.57 (Red Hat Enterprise Linux) OpenSSL/3.0.7 > mod_wsgi/5.0.0 Python/3.12 configured -- resuming normal operations > > [Tue Oct 15 15:10:34.246702 2024] [mpm_event:info] [pid 568866:tid 568866] > AH00490: Server built: Aug 5 2024 00:00:00 > > [Tue Oct 15 15:10:34.246719 2024] [core:notice] [pid 568866:tid 568866] > AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' > > [Tue Oct 15 15:10:34.248317 2024] [wsgi:info] [pid 568871:tid 568871] > mod_wsgi (pid=568871): Python home /opt/miniforge3/envs/chap. > > [Tue Oct 15 15:10:34.248321 2024] [wsgi:info] [pid 568872:tid 568872] > mod_wsgi (pid=568872): Python home /opt/miniforge3/envs/dataexplorer. > > [Tue Oct 15 15:10:34.248377 2024] [wsgi:info] [pid 568871:tid 568871] > mod_wsgi (pid=568871): Initializing Python. > > [Tue Oct 15 15:10:34.248387 2024] [wsgi:info] [pid 568872:tid 568872] > mod_wsgi (pid=568872): Initializing Python. > > [Tue Oct 15 15:10:34.249092 2024] [http2:info] [pid 568874:tid 568874] > h2_workers: created with min=25 max=37 idle_ms=600000 > > [Tue Oct 15 15:10:34.250137 2024] [wsgi:info] [pid 568874:tid 568874] > mod_wsgi (pid=568874): Initializing Python. > > [Tue Oct 15 15:10:34.256663 2024] [http2:info] [pid 568875:tid 568875] > h2_workers: created with min=25 max=37 idle_ms=600000 > > [Tue Oct 15 15:10:34.257760 2024] [wsgi:info] [pid 568875:tid 568875] > mod_wsgi (pid=568875): Initializing Python. > > [Tue Oct 15 15:10:34.258652 2024] [http2:info] [pid 568873:tid 568873] > h2_workers: created with min=25 max=37 idle_ms=600000 > > [Tue Oct 15 15:10:34.259732 2024] [wsgi:info] [pid 568873:tid 568873] > mod_wsgi (pid=568873): Initializing Python. > > [Tue Oct 15 15:10:34.262413 2024] [wsgi:info] [pid 568874:tid 568874] > mod_wsgi (pid=568874): Attach interpreter ''. > > [Tue Oct 15 15:10:34.271322 2024] [wsgi:info] [pid 568873:tid 568873] > mod_wsgi (pid=568873): Attach interpreter ''. > > [Tue Oct 15 15:10:34.277995 2024] [wsgi:info] [pid 568875:tid 568875] > mod_wsgi (pid=568875): Attach interpreter ''. > > [Tue Oct 15 15:10:34.279542 2024] [wsgi:info] [pid 568874:tid 568874] > mod_wsgi (pid=568874): Imported 'mod_wsgi'. > > [Tue Oct 15 15:10:34.288265 2024] [wsgi:info] [pid 568873:tid 568873] > mod_wsgi (pid=568873): Imported 'mod_wsgi'. > > [Tue Oct 15 15:10:34.296894 2024] [wsgi:info] [pid 568875:tid 568875] > mod_wsgi (pid=568875): Imported 'mod_wsgi'. > > [Tue Oct 15 15:10:49.266694 2024] [http2:info] [pid 569108:tid 569108] > h2_workers: created with min=25 max=37 idle_ms=600000 > > [Tue Oct 15 15:10:49.267916 2024] [wsgi:info] [pid 569108:tid 569108] > mod_wsgi (pid=569108): Initializing Python. > > [Tue Oct 15 15:10:49.281851 2024] [wsgi:info] [pid 569108:tid 569108] > mod_wsgi (pid=569108): Attach interpreter ''. > > [Tue Oct 15 15:10:49.298865 2024] [wsgi:info] [pid 569108:tid 569108] > mod_wsgi (pid=569108): Imported 'mod_wsgi'. > > [Tue Oct 15 15:10:50.264356 2024] [core:notice] [pid 568866:tid 568866] > AH00051: child pid 568872 exit signal Segmentation fault (11), possible > coredump in /etc/httpd > > [Tue Oct 15 15:10:50.264458 2024] [wsgi:info] [pid 568866:tid 568866] > mod_wsgi (pid=568872): Process 'dataexplorer' has died, deregister and > restart it. > > [Tue Oct 15 15:10:50.264466 2024] [wsgi:info] [pid 568866:tid 568866] > mod_wsgi (pid=568872): Process 'dataexplorer' terminated by signal 11 > > [Tue Oct 15 15:10:50.264473 2024] [wsgi:info] [pid 568866:tid 568866] > mod_wsgi (pid=568872): Process 'dataexplorer' has been deregistered and > will no longer be monitored. > > [Tue Oct 15 15:10:50.265514 2024] [wsgi:info] [pid 569173:tid 569173] > mod_wsgi (pid=569173): Starting process 'dataexplorer' with uid=36143, > gid=48 and threads=1. > > [Tue Oct 15 15:10:50.268450 2024] [wsgi:info] [pid 569173:tid 569173] > mod_wsgi (pid=569173): Python home /opt/miniforge3/envs/dataexplorer. > > [Tue Oct 15 15:10:50.268513 2024] [wsgi:info] [pid 569173:tid 569173] > mod_wsgi (pid=569173): Initializing Python. > > [Tue Oct 15 15:10:51.265981 2024] [core:notice] [pid 568866:tid 568866] > AH00051: child pid 569173 exit signal Segmentation fault (11), possible > coredump in /etc/httpd > > [Tue Oct 15 15:10:51.266037 2024] [wsgi:info] [pid 568866:tid 568866] > mod_wsgi (pid=569173): Process 'dataexplorer' has died, deregister and > restart it. > > [Tue Oct 15 15:10:51.266044 2024] [wsgi:info] [pid 568866:tid 568866] > mod_wsgi (pid=569173): Process 'dataexplorer' terminated by signal 11 > > [Tue Oct 15 15:10:51.266050 2024] [wsgi:info] [pid 568866:tid 568866] > mod_wsgi (pid=569173): Process 'dataexplorer' has been deregistered and > will no longer be monitored. > > [Tue Oct 15 15:10:51.267079 2024] [wsgi:info] [pid 569188:tid 569188] > mod_wsgi (pid=569188): Starting process 'dataexplorer' with uid=36143, > gid=48 and threads=1. > > [Tue Oct 15 15:10:51.269694 2024] [wsgi:info] [pid 569188:tid 569188] > mod_wsgi (pid=569188): Python home /opt/miniforge3/envs/dataexplorer. > > [Tue Oct 15 15:10:51.269761 2024] [wsgi:info] [pid 569188:tid 569188] > mod_wsgi (pid=569188): Initializing Python. > > > dataexplorer log: > > [Tue Oct 15 15:10:34.267712 2024] [wsgi:info] [pid 568872:tid 568872] > mod_wsgi (pid=568872): Attach interpreter ''. > > [Tue Oct 15 15:10:48.949190 2024] [wsgi:info] [pid 568872:tid 568981] > [remote 10.159.64.61:59510] mod_wsgi (pid=568872, process='dataexplorer', > application=''): Loading Python script file > '/var/www/vhosts/hydrosource-2/wsgi.py'. > > [Tue Oct 15 15:10:49.636199 2024] [wsgi:error] [pid 568875:tid 569031] > [client 10.159.64.61:59510] Truncated or oversized response headers > received from daemon process 'dataexplorer': > /var/www/vhosts/hydrosource-2/wsgi.py > > [Tue Oct 15 15:10:50.282619 2024] [wsgi:info] [pid 569173:tid 569173] > mod_wsgi (pid=569173): Attach interpreter ''. > > [Tue Oct 15 15:10:50.293954 2024] [wsgi:info] [pid 569173:tid 569176] > [remote 10.159.64.61:59511] mod_wsgi (pid=569173, process='dataexplorer', > application=''): Loading Python script file > '/var/www/vhosts/hydrosource-2/wsgi.py'. > > [Tue Oct 15 15:10:50.956465 2024] [wsgi:error] [pid 568874:tid 568980] > [client 10.159.64.61:59511] Truncated or oversized response headers > received from daemon process 'dataexplorer': > /var/www/vhosts/hydrosource-2/wsgi.py, referer: > https://dataexplorer.hydroclimate-dev.ornl.gov/ > > [Tue Oct 15 15:10:51.281032 2024] [wsgi:info] [pid 569188:tid 569188] > mod_wsgi (pid=569188): Attach interpreter ''. > > > Given the segmentation faults, I do a core dump and then use gdb to look > at the problem. To be honest, I don't really know what I'm looking at here. > If I do "bt", it prints out a lot more stuff, but I can't make heads or > tails of it. This is the output without backtracing with "bt". > > *GNU gdb (GDB) Red Hat Enterprise Linux 10.2-13.el9* > > Copyright (C) 2021 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later < > http://gnu.org/licenses/gpl.html> > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. > > Type "show copying" and "show warranty" for details. > > This GDB was configured as "x86_64-redhat-linux-gnu". > > Type "show configuration" for configuration details. > > For bug reporting instructions, please see: > > <https://www.gnu.org/software/gdb/bugs/>. > > Find the GDB manual and other documentation resources online at: > > <http://www.gnu.org/software/gdb/documentation/>. > > > For help, type "help". > > Type "apropos word" to search for commands related to "word"... > > Reading symbols from /usr/sbin/httpd... > > Reading symbols from .gnu_debugdata for /usr/sbin/httpd... > > (No debugging symbols found in .gnu_debugdata for /usr/sbin/httpd) > > > warning: Can't open file /dev/zero (deleted) during file-backed mapping > note processing > > [New LWP 568981] > > [New LWP 568872] > > [New LWP 568979] > > [New LWP 568977] > > [Thread debugging using libthread_db enabled] > > Using host libthread_db library "/lib64/libthread_db.so.1". > > Core was generated by `(wsgi:dataexplo -DFOREGROUND'. > > Program terminated with signal SIGSEGV, Segmentation fault. > > #0 PyObject_GetAttr (v=0x7fa2e8a44130, name=0x7fa2eae33c20 > <_PyRuntime+45280>) at /usr/local/src/conda/python-3.12.6/Include/object.h > :968 > > 968 /usr/local/src/conda/python-3.12.6/Include/object.h: No such file or > directory. > > [Current thread is 1 (Thread 0x7fa2e9425640 (LWP 568981))] > > Missing separate debuginfos, use: dnf debuginfo-install > httpd-core-2.4.57-11.el9_4.1.x86_64 > > > It seems like it's routing to an incorrect python install somehow. > /usr/local/src/conda > is not a path that exists on this machine, so the error makes sense, but I > don't know what it's trying to do on this line. > > I have attempted to set header-buffer-size on my WSGIProcessDaemon > directives but haven't had any luck with that. If I remove the offending > packages (things that have to do with SQL like flask-sqlalchemy, psycopg2, > geoalchemy2, etc) then the apps will run without crashing, but will > obviously give me errors about being unable to import the packages I > removed since they're needed in my app. I'm really not sure how to move > forward on this. Is there something obviously wrong in my configuration? Do > I need to install mod_wsgi in each virtual environment instead of the base > environment (and if so, then how would LoadModule work in Apache?). > > > Lastly, here's my wsgi.py script file (each app has the same one). I am > using blueprints in my Flask app, so I import the create_app function to > generate the application. > import sys > from api import create_app > from os import getcwd > > sys.path.insert(0, getcwd() + "/api") > sys.path.append(getcwd() + "/api/models") > > application = create_app() > > Any insight on how to debug this would be very helpful. I have reviewed > the debugging technique docs, but I really feel lost with the gnu debugger. > > Thanks for your time! > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/b6af7e32-21b7-4f04-9e5b-ab1e8cfaa7dan%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/b6af7e32-21b7-4f04-9e5b-ab1e8cfaa7dan%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/a96269ef-2e4a-4c4c-9a1f-877cf9e65e04n%40googlegroups.com.
