I was actually thinking of whether to ask if using New Relic. I didn't because the symptom in New Relic agent which would have explained it (import in a new thread created while parent thread doing agent initialisation was still doing an import), was fixed long ago, and from what I remember actually even before the version you are running. Also, technically that problem shouldn't have occurred under mod_wsgi if the agent was initialised in WSGI script file. So I am not sure what specific issue you hit on the New Relic agent, but good to know you solved it.
Graham > On 15 Mar 2017, at 12:03 AM, David <[email protected]> wrote: > > Hi Graham, > > Thank you for all the help you've provided, it's been crucial to fixing my > installation and configuration of mod_wsgi. I managed to get a little help > from a developer who proceeded to test the wsgi.py file we utilise and after > a little debugging he found out that the issue stemmed from the inclusion of > newrelic within the script. I proceeded to upgrade the package from version > 2.16.0.12 to 2.80.1.61 which fixed the issue completely and page load times > went from ~40 to ~5 seconds . I also checked the Apache access logs where I > saw the line that points toward mod_wsgi loading the WSGI script. > > Many Thanks, > David > > On Monday, 6 March 2017 23:02:46 UTC, Graham Dumpleton wrote: > If you have info set for LogLevel, you should see a line like: > > mod_wsgi (pid=85737, process='localhost:8000', application=''): Loading WSGI > script '/tmp/mod_wsgi-localhost:8000:502/handler.wsgi'. > > Are you saying you do not see that in the error log? > > When pre-loading is setup, you should see that when any process is first > loaded. If you are only see it on the first request, that isn't right. > > So take note of where you see that. > > Wait a full minute before doing the first request so can see whether it is > the global import of the WSGI script file on process start which takes a long > time and delaying first request because initial import still not finished. > > Graham > >> On 7 Mar 2017, at 12:48 AM, David <dcw...@ <>gmail.com <http://gmail.com/>> >> wrote: >> >> Hi Graham, >> >> I had mistyped the python-home directory location when I submitted my >> response yesterday. The python-home configuration setting in the virtual >> host file is indeed correct, sorry for the confusion. I've looked into and >> fixed the "No module named site" issue thanks to the help of your advice on >> another post using the commands seen below. >> >> ./configure --with-python=/usr/local/bin/python2.7 >> LD_RUN_PATH=/usr/local/lib make >> make install >> >> The installation procedure that configured it incorrectly before was via >> configuration management software which I will now update. The overall >> configuration is loadable and will respond to requests so it's great to have >> everything setup as correctly as it can be now. >> >> I have still to find out what the issue is regarding slow response times >> when a request is made to the application after the processes have been >> created. I have included snippets from Apaches logs to hopefully provide >> some further information in case it shines a light on an issue that I can't >> see for myself. >> >> The logs show me restarting the httpd service using the "systemctl restart >> httpd" command at 13:32:51. I then make a request at 13:33:03 at which point >> I wait for roughly 30 seconds before I receive the response from the server >> showing the page requested. It should be noted that the log entry in the >> xxx.com.access.log file at 13:33:03 didn't actually appear until 13:33:33 or >> so, so it's a bit misleading at a glance as well as the wait time can be >> between 30 - 60 seconds but in the example below it is roughly 30 seconds. >> >> ==> /var/log/httpd/error_log <== >> [Mon Mar 06 13:32:51.981992 2017] [core:info] [pid 11691:tid >> 140191937927296] AH00096: removed PID file /run/httpd/httpd.pid (pid=11691) >> [Mon Mar 06 13:32:51.982022 2017] [mpm_worker:notice] [pid 11691:tid >> 140191937927296] AH00296: caught SIGWINCH, shutting down gracefully >> [Mon Mar 06 13:32:52.999652 2017] [wsgi:info] [pid 11691:tid >> 140191937927296] mod_wsgi (pid=11692): Process 'xxx.com <http://xxx.com/>' >> to be deregistered, as server is restarting or being shutdown. >> [Mon Mar 06 13:32:52.999708 2017] [wsgi:info] [pid 11691:tid >> 140191937927296] mod_wsgi (pid=11692): Process 'xxx.com <http://xxx.com/>' >> has been deregistered and will no longer be monitored. >> [Mon Mar 06 13:32:54.583209 2017] [auth_digest:notice] [pid 11996:tid >> 140146629888128] AH01757: generating secret for digest authentication ... >> [Mon Mar 06 13:32:54.588361 2017] [mpm_worker:notice] [pid 11996:tid >> 140146629888128] AH00292: Apache/2.4.6 (CentOS) mod_wsgi/4.5.14 Python/2.7 >> configured -- resuming normal operations >> [Mon Mar 06 13:32:54.588382 2017] [mpm_worker:info] [pid 11996:tid >> 140146629888128] AH00293: Server built: Nov 14 2016 18:04:44 >> [Mon Mar 06 13:32:54.588397 2017] [core:notice] [pid 11996:tid >> 140146629888128] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' >> [Mon Mar 06 13:32:54.602488 2017] [wsgi:info] [pid 11997:tid >> 140146629888128] mod_wsgi (pid=11997): Starting process 'xxx.com >> <http://xxx.com/>' with uid=1001, gid=1003 and threads=2. >> [Mon Mar 06 13:32:54.602891 2017] [wsgi:info] [pid 11997:tid >> 140146629888128] mod_wsgi (pid=11997): Python home /srv/django/xxx.com >> <http://xxx.com/>. >> [Mon Mar 06 13:32:54.602915 2017] [wsgi:info] [pid 11997:tid >> 140146629888128] mod_wsgi (pid=11997): Initializing Python. >> [Mon Mar 06 13:32:54.616328 2017] [wsgi:info] [pid 11998:tid >> 140146629888128] mod_wsgi (pid=11998): Starting process 'xxx.com >> <http://xxx.com/>' with uid=1001, gid=1003 and threads=2. >> [Mon Mar 06 13:32:54.616745 2017] [wsgi:info] [pid 11998:tid >> 140146629888128] mod_wsgi (pid=11998): Python home /srv/django/xxx.com >> <http://xxx.com/>. >> [Mon Mar 06 13:32:54.616767 2017] [wsgi:info] [pid 11998:tid >> 140146629888128] mod_wsgi (pid=11998): Initializing Python. >> >> ==> /var/log/httpd/xxx.com.access.log <== >> xxx.com <http://xxx.com/> 222.222.222.222 "111.111.111.111" - >> [06/Mar/2017:13:33:03 +0000] "GET / HTTP/1.1" 200 66126 "-" "Mozilla/5.0 >> (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) >> Chrome/56.0.2924.87 Safari/537.36" "-" >> xxx.com <http://xxx.com/> 222.222.222.222 "111.111.111.111" - >> [06/Mar/2017:13:33:37 +0000] "GET /request2/ HTTP/1.1" 200 17570 >> "https://xxx.com/ <https://xxx.com/>" "Mozilla/5.0 (Macintosh; Intel Mac OS >> X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 >> Safari/537.36" "-" >> >> Thanks, >> David >> >> On Monday, 6 March 2017 00:06:45 UTC, Graham Dumpleton wrote: >> >>> On 6 Mar 2017, at 10:10 AM, David <[email protected] <>> wrote: >>> >>> Hi Graham, >>> >>> First off thank you very much for the support you have given, I really >>> appreciate the time you've taken to help me. I read over the changes you >>> kindly pointed out and have now amended the virtual host file to use the >>> following configuration. >>> >>> virtual host file >>> WSGIRestrictEmbedded On >>> <VirtualHost *:80> >>> >>> ... >>> >>> WSGIPassAuthorization On >>> >>> WSGIDaemonProcess xxx.com <http://xxx.com/> user=django group=admin >>> processes=2 threads=2 display-name=xxx.com <http://xxx.com/> >>> python-home=/srv/django/xxx >> >> Did you get the argument to python-home wrong here. Doesn't match below. >> >>> WSGIScriptAlias / /srv/django/xxx.com/xxx/xxx/wsgi.py >>> <http://xxx.com/xxx/xxx/wsgi.py> process-group=xxx.com <http://xxx.com/> >>> application-group=%{GLOBAL} >>> >>> <Directory /srv/django/xxx.com/xxx/ <http://xxx.com/xxx/>> >>> Order allow,deny >>> Allow from all >>> </Directory> >>> >>> </VirtualHost> >>> >>> >>> Once I had this configuration in place I received the following error. >>> >>> [wsgi:info] [pid 19595:tid 140504746649728] mod_wsgi (pid=19595): Python >>> home /srv/django/xxx.com <http://xxx.com/>. >>> [wsgi:info] [pid 19595:tid 140504746649728] mod_wsgi (pid=19595): >>> Initializing Python. >>> ImportError: No module named site >> >> When using python-home, this can indicate that your mod_wsgi may not be >> compiled with the same Python base installation as the virtual environment >> was created from. >> >> >> http://modwsgi.readthedocs.io/en/develop/user-guides/checking-your-installation.html#python-installation-in-use >> >> <http://modwsgi.readthedocs.io/en/develop/user-guides/checking-your-installation.html#python-installation-in-use> >> >> or that is not picking up correct Python shared library. >> >>> I double checked my installation of mod_wsgi and confirmed that it was >>> indeed configured against my alternative installed python version that I >>> had wanted and then checked my virtual environment was using the same >>> alternative installed Python version and it was. >> >> The problem is where your system Python installation is same major/minor >> version as an alternate Python installation. If you didn't override which >> Python library was being used, it could be picking up system Python library >> and not that for alternative installation. >> >>> I then decided to pip install the mod_wsgi package within the virtual >>> environment and then modified the "LoadModules" path for mod_wsgi to that >>> of the virtual environments as can be seen below. >>> >>> LoadModule wsgi_module >>> /srv/django/xxx.com/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so >>> >>> <http://xxx.com/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so> >> If using that method, ensure you use the output of running: >> >> mod_wsgi-express module-config >> >>> This alleviated the issue of "No module named site" >> >> When using pip install method, it tries to ensure that correct Python shared >> library is linked explicitly into the .so file. To be extra safe, don't >> calculate path of .so yourself. Include all the output from running: >> >> mod_wsgi-express module-config >> >> into Apache configuration. This can include a LoadFile line for force loads >> correct Python library. >> >> Also includes a global WSGIPythonHome to ensure using correct Python runtime >> directory. >> >>> but it still left me page load times of roughly ~1 minute upon request. >> >> Ensure you have LogLevel in Apache set to 'info' and not 'warn' or 'error'. >> >> The 'info' level will show when mod_wsgi loads WSGI script file initially so >> can verify occurs on process start. >> >>> I restarted the server and waited roughly a minute before requesting a >>> page, hoping to verify that the application had loaded into the processes >>> by then but instead of seeing a response of 3 seconds or less I got ~1 >>> minute. This is leaving me to think that the application isn't being loaded >>> into the processes until a request has been made whereas I though the >>> configuration allowed the application to preload before a request was made? >>> >>> Is there any where else to look at this point that I may be overlooking, I >>> have a feeling that it may be in and around the area that led me to pip >>> install mod_wsgi instead of using my source installed version? >> >> Check things about, but also enable logging at 'info' level so can see when >> things are loaded. Will then work out where to go next. >> >>> Many Thanks, >>> David >>> >>> On Saturday, 4 March 2017 22:21:36 UTC, Graham Dumpleton wrote: >>> >>>> On 5 Mar 2017, at 7:42 AM, David <dcw...@ <>gmail.com <http://gmail.com/>> >>>> wrote: >>>> >>>> I've found an issue in heavily increased loading time when mod_wsgi loads >>>> the application for the first time. This has started since since upgrading >>>> from my old setup of CentOS 6, Python 2.7.5, Apache 2.2.15, and mod_wsgi >>>> 3.4 to my new setup of CentOS 7, Python 2.7.13, Apache 2.4.6, and mod_wsgi >>>> 4.5.13. >>>> >>>> The change has caused the initial loading time of the application into >>>> memory from ~5 seconds to ~1 minute. I have followed the "Checking Your >>>> Installation" as much as I could, coming up with a blank as to what could >>>> be causing this increase. >>>> >>>> The mod_wsgi and Apache configuration is the same across both the older >>>> and new setup, configured to use the worker mpm module running in daemon >>>> mode. Please find the configuration of both the virtualhost file and the >>>> worker configuration found in the main httpd.conf file below. >>>> >>>> httpd.conf >>>> <IfModule mpm_worker_module> >>>> StartServers 16 >>>> MaxClients 250 >>>> MinSpareThreads 10 >>>> MaxSpareThreads 10 >>>> ThreadsPerChild 25 >>>> MaxRequestsPerChild 0 >>>> </IfModule> >>>> >>>> >>>> virtualhost configuration >>>> >>>> WSGISocketPrefix run/wsgi >>>> WSGILazyInitialization On >>> >>> This is the default, you don't need this directive. >>> >>>> WSGIRestrictEmbedded Off >>> >>> You want to set this to On, not Off. >>> >>>> <VitualHost *:80> >>>> >>>> ... >>>> >>>> WSGIPassAuthorization On >>>> WSGIDaemonProcess xxx.com <http://xxx.com/> user=django group=admin >>>> processes=2 threads=2 display-name=xxx.com <http://xxx.com/> >>>> python-path=/srv/django/xxx/lib/python2.7/site-packages >>> >>> Use: >>> >>> WSGIDaemonProcess xxx.com <http://xxx.com/> user=django group=admin >>> processes=2 threads=2 display-name=xxx.com <http://xxx.com/> >>> python-home=/srv/django/xxx >>> >>> You should not use python-path for setting up a virtual environment, use >>> python-home instead. >>> >>>> WSGIProcessGroup xxx.com <http://xxx.com/> >>> You don't need WSGIProcessGroup if set process-group as option to >>> WSGIScriptAlias. See below. >>> >>>> WSGIScriptAlias / /srv/django/xxx/xxx/xxx/wsgi.py >>>> WSGIImportScript /srv/django/xxx/xxx/xxx/wsgi.py process-group=xxx.com >>>> <http://xxx.com/> application-group=xxx.com <http://xxx.com/> >>> It is better to use: >>> >>> WSGIScriptAlias / /srv/django/xxx/xxx/xxx/wsgi.py process-group=xxx.com >>> <http://xxx.com/> application-group=%{GLOBAL} >>> >>> to force preloading. Remove the WSGIImportScript directive. >>> >>>> <Directory /srv/django/xxx/xxx/> >>>> WSGIApplicationGroup xxx.com <http://xxx.com/> >>> Since you only have the one application and are using daemon mode, you >>> should use: >>> >>> WSGIApplicationGroup %{GLOBAL} >>> >>> But then you don't need this directive at all if you are forcing >>> pre-loading and have supplied application-group option to WSGIScriptAlias. >>> See above. >>> >>>> WSGIScriptReloading On >>> >>> This is the default, you don't need this directive. >>> >>>> Order allow,deny >>>> Allow from all >>>> </Directory> >>>> >>>> ... >>>> >>>> </VirtualHost> >>>> >>>> >>>> Any help would be greatly appreciated and if anyone can point me in the >>>> right direction as to how to debug this issue further I will gladly oblige >>>> to the best of my ability! >>> >>> Make those changes and see how it goes. >>> >>> Because you have been force pre-loading, the WSGI application should be >>> getting loaded when daemon processes start, not on first request. >>> >>> Graham >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>. >>> To post to this group, send email to mod...@ <>googlegroups.com >>> <http://googlegroups.com/>. >>> Visit this group at https://groups.google.com/group/modwsgi >>> <https://groups.google.com/group/modwsgi>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <>. >> To post to this group, send email to [email protected] <>. >> Visit this group at https://groups.google.com/group/modwsgi >> <https://groups.google.com/group/modwsgi>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
