Done. Will tell you when error appears again.
On Monday, October 26, 2015 at 12:46:41 PM UTC+2, Graham Dumpleton wrote:
>
> The next thing to do is set LogLevel to info and get the better logging
> from Apache error log for when the problems arise as already indicated.
>
> Graham
>
> On 26 Oct 2015, at 9:42 pm, Paul Royik <distan...@gmail.com <javascript:>>
> wrote:
>
> ServerRoot "/home/simamura/webapps/django_math/apache2"
>
> LoadModule alias_module modules/mod_alias.so
> LoadModule authz_core_module modules/mod_authz_core.so
> LoadModule dir_module modules/mod_dir.so
> LoadModule env_module modules/mod_env.so
> LoadModule log_config_module modules/mod_log_config.so
> LoadModule mime_module modules/mod_mime.so
> LoadModule rewrite_module modules/mod_rewrite.so
> LoadModule setenvif_module modules/mod_setenvif.so
> LoadModule wsgi_module modules/mod_wsgi.so
> LoadModule unixd_module modules/mod_unixd.so
>
> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\"
> \"%{User-Agent}i\"" combined
> CustomLog /home/simamura/logs/user/access_django_math.log combined
> ErrorLog /home/simamura/logs/user/error_django_math.log
>
> Listen 30821
> KeepAlive Off
> SetEnvIf X-Forwarded-SSL on HTTPS=1
> ServerLimit 5
> StartServers 1
> MaxRequestWorkers 60
> MinSpareThreads 1
> MaxSpareThreads 3
> ThreadsPerChild 12
>
> WSGIApplicationGroup %{GLOBAL}
> WSGIDaemonProcess django_math processes=5 threads=12
> python-path=/home/simamura/webapps/django_math:/home/simamura/webapps/django_math/mathsite:/home/simamura/webapps/django_math/lib/python2.7
> WSGIProcessGroup django_math
> WSGIRestrictEmbedded On
> WSGILazyInitialization On
> WSGIScriptAlias /
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
>
> I changed a bit threadlimit as webfaction advised.
> What should I do next?
>
> On Monday, October 26, 2015 at 3:13:58 AM UTC+2, Graham Dumpleton wrote:
>
> Can you show me what your mod_wsgi configuration in Apache configuration
> file actually is? Ensure that the WSGIDaemonProcess directive is included.
>
> Ensure that the LogLevel directive in Apache is set to ‘info’ and not
> ‘warn’ and provide the actual log messages from the Apache error log from
> when it occurs, including a dozen lines messages before and after so can
> see context. Prior and followup messages can contain important information
> which you might not think is helpful.
>
> Graham
>
> On 23 Oct 2015, at 8:27 pm, Paul Royik <distan...@gmail.com> wrote:
>
> The most long running request is 12 seconds.
> I tried to set request-timeout, but it didn't help.
>
> On Friday, October 23, 2015 at 1:25:13 AM UTC+3, Graham Dumpleton wrote:
>
> No MaxRequestWorkers will not necessarily help. It doesn’t control how
> many processes/threads are available in the daemon processes used by
> mod_wsgi. Even changing the processes/threads for the daemon processes is
> only masking or delaying the occurrence of the issue. You need to work out
> the underlying cause.
>
> I would suggest integrating the example code found in:
>
>
> https://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_Python_Stack_Traces
>
> This will give you a way of being able to get Python stack traces out of
> the application as to what it is doing when it appears to get stuck.
>
> If you have a new enough mod_wsgi version, you can also try setting on
> WSGIDaemonProcess the option:
>
> request-timeout=60
>
> This will set a timeout on how long requests are allowed to run. If
> exceeded then the process will be forcibly restarted.
>
> This will recover from processes which are locked up and then cause who
> Apache to run out of capacity.
>
> Note that 60 means 60 seconds for a single request in case of single
> thread daemon process.
>
> Things get more complicated for multiple process and the time value
> relates to average consumed time across all active requests. So technically
> a single request can run longer than the timeout.
>
> As part of the process of restarting the process, mod_wsgi will attempt to
> dump Python stack traces as to where the code may have been running at that
> time, and so can work similar to the manual method mentioned in that link.
>
> If you play with this option, set the timeout to be twice what your
> longest request would take. Although, if you have very long running
> requests, what to set it to may be a problem in itself.
>
> Graham
>
> On 23 Oct 2015, at 4:24 am, Paul Royik <distan...@gmail.com> wrote:
>
> I set WSGIApplicationGroup, but it doesn't help.
>
> I used sympy all year, but problems begin only now. Maybe when I upgraded
> to Apache 2.4
>
> Anyway, in log file I got
> [Thu Oct 22 17:08:55.697080 2015] [mpm_worker:error] [pid 658868:tid
> 140669018494784] AH00286: server reached MaxRequestWorkers setting,
> consider raising the MaxRequestWorkers setting
> [Thu Oct 22 17:09:26.399723 2015] [wsgi:error] [pid 667167:tid
> 140668926658304] [client 127.0.0.1:58424] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%282x%5E4%29%2F%28x%5E4-1%29&var=&a=2&b=3&steps=on
>
> Will it help if I raise MaxRequestWorkers? What is the readonable number?
>
> On Thursday, October 22, 2015 at 2:13:07 PM UTC+3, Graham Dumpleton wrote:
>
> No you cannot change the network keep alive timeout, or not readily and it
> isn’t something you should ever do. It is quite possibly nothing to do with
> that anyway.
>
> As to WSGIApplicationGroup then it may help as I already outlined. Any
> reason you haven’t tried it as it is recommended practice to use it?
>
> What exactly are you using simpy for anyway. Looking at the documentation
> for it it does various stuff with sub processes. Heavy or complicated use
> of sub processes from inside of the context of a web application,
> especially under Apache/mod_wsgi, is generally a bad idea. This is because
> it may not play well with Apache’s process model and so something like
> simpy may not work properly or could even interfere with the process
> management of Apache or mod_wsgi.
>
> Graham
>
> On 22 Oct 2015, at 4:41 pm, Paul Royik <distan...@gmail.com> wrote:
>
> Problem appeared again.
> So, I will answer your questions.
> I use third-party libraries sympy and sage. Didn't have any network issues.
> Actually this happened when I sleep.
> Exact error time is 2 hours 11 minutes (this value is always the same).
>
> So, can I lower network keep alive timeout value?
>
> If I set WSGIApplicationGroup %{GLOBAL}, will this help?
>
> Thank you.
>
> On Tuesday, October 13, 2015 at 1:39:31 PM UTC+3, Graham Dumpleton wrote:
>
> The messages indicate that all the threads in the daemon process group
> were blocked.
>
> Initially new requests will still get added to the socket listener queue
> for the daemon processes, which has a capacity for 100 queued requests by
> default. Because though the daemon process wasn’t actually accepted them
> and removing them from the queue, on the Apache child worker side where
> they were being proxied, they would timeout. That accounts for the message:
>
> Timeout when reading response headers from daemon process
>
> When that 100 queue depth was eventually exceeded, no more requests could
> be queued up. This then resulted in attempts to connect to the daemon
> processes failing outright, which is what the error:
>
> Unable to connect to WSGI daemon process
>
> is all about.
>
> First up I would suggest you also ensure you are setting:
>
> WSGIApplicationGroup %{GLOBAL}
>
> This forces the application to run the in the main interpreter context of
> the daemon process groups.
>
> This avoids problems with third party extension modules for Python that
> haven’t been implemented correctly so as to be able to be used in sub
> interpreter contexts. These can only work in the main interpreter context.
>
> This is often a cause of thread deadlocks.
>
> That said, that you say it lasts 2 hours and then disappears is
> interesting.
>
> This is because 2 hours is the network keep alive timeout value.
>
> So what may be happening is request threads are contacting some backend
> service and due to some network/router issue, the connection is being
> broken. In some circumstances, depending on what software is doing with the
> connection, such a condition may not be detected until the 2 hour network
> timeout occurs.
>
> In light of that, what backend services does your application talk to and
> would that traffic traverse over any switching equipment. Have you been
> having any network issues.
>
> Graham
>
>
> On 12 Oct 2015, at 11:55 pm, Paul Royik <distan...@gmail.com> wrote:
>
> Hi Graham and everyone.
>
> I have a site that runs heavy math calculations using external math
> library.
>
> I already wrote about it nearly half a year ago. In that case there was
> timeout issue, but now I've fixed it.
>
>
> But there is another issue.
> Nearly the same time I got 503 error: Timeout when reading response
> headers from daemon process.
> This lasts for approximately 2 hours and then 503 error disappears.
>
> I use Django 1.8.4 with mod_wsgi 4.4.13.
>
> What configurations should I use in httpd.conf to avoid this error.
> How to queue requests?
> What is the number of sumultaneous requests?
> ....
>
>
> What I want to do is avoid 503 error, limit number of simultaneous
> requests and queue other requests.
>
>
> My log file:
> [Mon Oct 12 11:47:09.445618 2015] [wsgi:error] [pid 10210:tid
> 139829293606656] [client 127.0.0.1:45625] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
> [Mon Oct 12 11:47:14.432179 2015] [wsgi:error] [pid 10210:tid
> 139829390120704] [client 127.0.0.1:45632] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/?f=tan%28x%2F2%29+%2B+%281%2F%28x%5E2%2B4%29%29&a=0&b=2&n=8&steps=on
> [Mon Oct 12 11:47:20.201940 2015] [wsgi:error] [pid 10210:tid
> 139829400610560] [client 127.0.0.1:45700] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on
> [Mon Oct 12 11:47:20.608201 2015] [wsgi:error] [pid 10210:tid
> 139829283116800] [client 127.0.0.1:45704] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
> [Mon Oct 12 11:48:08.984176 2015] [wsgi:error] [pid 10210:tid
> 139829304096512] [client 127.0.0.1:45707] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/
> [Mon Oct 12 11:48:09.505969 2015] [wsgi:error] [pid 10210:tid
> 139829293606656] [client 127.0.0.1:45730] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on
> [Mon Oct 12 11:48:14.492629 2015] [wsgi:error] [pid 10210:tid
> 139829390120704] [client 127.0.0.1:45731] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
> [Mon Oct 12 11:48:20.262302 2015] [wsgi:error] [pid 10210:tid
> 139829400610560] [client 127.0.0.1:45732] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on
> [Mon Oct 12 11:48:20.668421 2015] [wsgi:error] [pid 10210:tid
> 139829283116800] [client 127.0.0.1:45737] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> https://www.google.co.uk/
> [Mon Oct 12 11:49:09.044532 2015] [wsgi:error] [pid 10210:tid
> 139829304096512] [client 127.0.0.1:45744] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> https://www.google.co.uk/
> [Mon Oct 12 11:49:09.566430 2015] [wsgi:error] [pid 10210:tid
> 139829293606656] [client 127.0.0.1:45789] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=sin%5E5+%28x%29&var=&steps=on
> [Mon Oct 12 11:49:14.552970 2015] [wsgi:error] [pid 10210:tid
> 139829390120704] [client 127.0.0.1:45795] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> https://www.google.co.uk/
> [Mon Oct 12 11:49:20.322179 2015] [wsgi:error] [pid 10210:tid
> 139829400610560] [client 127.0.0.1:45799] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/
> [Mon Oct 12 11:49:20.728784 2015] [wsgi:error] [pid 10210:tid
> 139829283116800] [client 127.0.0.1:45803] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on
> [Mon Oct 12 11:50:09.104426 2015] [wsgi:error] [pid 10210:tid
> 139829304096512] [client 127.0.0.1:45816] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=ln+x+&var=x&steps=on
> [Mon Oct 12 11:50:09.626783 2015] [wsgi:error] [pid 10210:tid
> 139829293606656] [client 127.0.0.1:45819] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%28y%5E%281%2F2%29%2F2%29%2B%281%2F%282*y%5E%281%2F2%29%29%29&var=&a=3&b=9
> [Mon Oct 12 11:50:14.613330 2015] [wsgi:error] [pid 10210:tid
> 139829390120704] [client 127.0.0.1:45820] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/
> [Mon Oct 12 11:50:20.382640 2015] [wsgi:error] [pid 10210:tid
> 139829400610560] [client 127.0.0.1:45832] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
> [Mon Oct 12 11:50:20.789229 2015] [wsgi:error] [pid 10210:tid
> 139829283116800] [client 127.0.0.1:45846] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> https://www.google.co.uk/
> [Mon Oct 12 11:51:09.164863 2015] [wsgi:error] [pid 10210:tid
> 139829304096512] [client 127.0.0.1:45848] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> https://www.google.co.uk/
> [Mon Oct 12 11:51:09.634602 2015] [wsgi:error] [pid 10210:tid
> 139829293606656] [client 127.0.0.1:45879] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> http://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&ved=0CFAQFjAJahUKEwje1pz16rzIAhXKGI4KHVrPAzA&url=http%3A%2F%2Fwww.emathhelp.net%2Fnotes%2Fcalculus-1%2Fdrawing-graphs-of-functions%2Fsteps-for-sketching-graph-of-function%2F&usg=AFQjCNGAABCyCB7rhd4yAI83Dd0NDLTrPw
> [Mon Oct 12 11:51:14.639873 2015] [wsgi:error] [pid 10210:tid
> 139829390120704] [client 127.0.0.1:45914] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> https://www.google.co.uk/
> [Mon Oct 12 11:51:20.443005 2015] [wsgi:error] [pid 10210:tid
> 139829400610560] [client 127.0.0.1:45918] Timeout when reading response
> headers from daemon process 'django_math':
> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
> https://www.google.com/
> [Mon Oct 12 11:51:20.849576 2015] [wsgi:error] [pid 10210:tid
> 139829283116800] [client
>
> ...
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to modwsgi+u...@googlegroups.com.
> To post to this group, send email to mod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to modwsgi+u...@googlegroups.com <javascript:>.
> To post to this group, send email to mod...@googlegroups.com <javascript:>
> .
> Visit this group at http://groups.google.com/group/modwsgi.
>
> ...
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to modwsgi+unsubscr...@googlegroups.com.
To post to this group, send email to modwsgi@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
