Actually this is why we had issues only recently: our config was a bit stupid and we had more or less the same values for all timeouts ; and we know that this specific request could take up to hours, so we had set high timeouts, including for shutdown and graceful ; except that we cleaned (but obviously not enough) the config and we removed these, which lead to kill long running requests if maximum-requests number has been reached.
For now I removed the maximum-requests setting - I don't think we still need it. Thanks again ! On Wednesday, November 25, 2015 at 11:23:03 AM UTC+1, Graham Dumpleton wrote: > > If you really can’t avoid use of maximum-requests for some reason, then in > more recent mod_wsgi versions you can also set a graceful-timeout option at > the same time. > > When graceful-timeout is set and maximum requests is reached, rather than > immediately stopping accepting of new requests and then forcing shutdown > after 5 seconds if requests don’t finish, then the process will continue to > accept requests for the period of the graceful timeout first. > > If during graceful timeout a point is reached where there are not active > requests the process will be shutdown immediately. If the graceful timeout > period expires then new requests will not be accepted and the maximum > shutdown timeout applied. > > The graceful timeout thus can be set much higher than the 5 second default > shutdown timeout to allow long requests more time to finish. Of course if > more long running requests come in, then you still end up with the same > issue with forced shutdown occurring. It does give you a greater chance > that shutdown will occur without interrupting requests. > > In mod_wsgi-express the default graceful timeout is 15 seconds. In > mod_wsgi alone, the graceful timeout isn’t set so only shutdown timeout is > applied. > > Graham > > On 25 Nov 2015, at 7:29 PM, Pierre-Nicolas Rigal <[email protected] > <javascript:>> wrote: > > Yes we do use maximum-requests ; IIRC, we introduced that a "long" time > ago (we were using apache 2.2 and mod_wsgi 3 if I'm right). > I remember we had high memory consumption from httpd and someone in the > team found somewhere (I'd bet stack overflow ;)) that maximum-requests > would solve the issue. > > Not sure exactly to remember the details (I'd bet on customers not > understanding that "free memory" is not the right indicator). > I'm changing the config right now. > > Thanks for the input ! > Pierre-Nicolas > > > > > On Wednesday, November 25, 2015 at 1:20:14 AM UTC+1, Graham Dumpleton > wrote: >> >> What do you have maximum-requests set to on WSGIDaemonProcess directive >> and why are you using it? >> >> The logs show: >> >> mod_wsgi (pid=16036): Maximum requests reached, triggering immediate >> shutdown 'django’. >> >> This means you are telling mod_wsgi that processes should be killed off >> and restarted every certain number of requests. >> >> This should not be used in production systems unless you have a very good >> need to use it and definitely not if you have any long running requests. >> >> The reason is that when such a restart is triggered, the process cannot >> wait indefinitely for any still running requests to complete as a new >> process will not be created until old one can be killed off. >> >> Thus any still running requests get at most 5 seconds of grace to >> complete. In that time no new requests are accepted by that process. >> >> You can see this occurring in: >> >> [Tue Nov 24 09:34:14.591027 2015] [wsgi:info] [pid 16037:tid >> 140593024513792] mod_wsgi (pid=16037): Maximum requests reached, triggering >> immediate shutdown 'django'. >> [Tue Nov 24 09:34:14.591327 2015] [wsgi:info] [pid 16037:tid >> 140593301862208] mod_wsgi (pid=16037): Shutdown requested 'django'. >> ... >> [Tue Nov 24 09:34:19.591853 2015] [wsgi:info] [pid 16037:tid >> 140592623019776] mod_wsgi (pid=16037): Aborting process 'django'. >> [Tue Nov 24 09:34:19.591938 2015] [wsgi:info] [pid 16037:tid >> 140592623019776] mod_wsgi (pid=16037): Exiting process 'django'. >> ... >> [Tue Nov 24 09:34:20.338880 2015] [wsgi:info] [pid 1961:tid >> 140593301862208] mod_wsgi (pid=16037): Process 'django' has died, >> deregister and restart it. >> [Tue Nov 24 09:34:20.339018 2015] [wsgi:info] [pid 1961:tid >> 140593301862208] mod_wsgi (pid=16037): Process 'django' has been >> deregistered and will no longer be monitored. >> >> The process was forced aborted as active requests didn’t complete in >> time. This would cause those long running requests to be interrupted and a >> 500 response would be seen by the user. >> >> So what problem are you trying to solve by using maximum-requests? There >> may be a better way with more recent versions of mod_wsgi. >> >> Graham >> >> On 25 Nov 2015, at 1:45 AM, Pierre-Nicolas Rigal <[email protected]> >> wrote: >> >> Hi Graham, >> >> thanks for your quick answer. >> I attached logs - the main error is: >> >> [Tue Nov 24 09:34:52.565669 2015] [wsgi:error] [pid 16056:tid >> 140592908560128] [client 213.196.133.30:50931] Truncated or oversized >> response headers received from daemon process 'django': >> /usr/local/filewave/django/filewave/apache/django.wsgi >> >> Anything rings a bell in the logs ? >> Thanks and Regards, >> Pierre-Nicolas >> >> >> On Sunday, November 22, 2015 at 11:47:36 PM UTC+1, Graham Dumpleton wrote: >>> >>> Can't provide an in depth reply right now, but set LogLevel directive in >>> Apache to 'info' and mod_wsgi will provide more information about when/why >>> processes are restarting. This may provide clue as to what is happening. >>> Capture what extra logs you can from around the 500 and provide them. >>> >>> Thanks. >>> >>> Graham >>> >>> On 23 Nov 2015, at 7:24 AM, Pierre-Nicolas Rigal <pierren...@ >>> filewave.com> wrote: >>> >>> Hello, >>> >>> I'm using apache 2.4.17 and mod_wsgi 4.4.21, and we have customers >>> reporting errors 500. Looking at the logs, we can see messages like: >>> >>> [Fri Nov 20 09:57:12.905868 2015] [wsgi:error] [pid 26230:tid >>> 140671746815744] [client 10.10.12.50:42501] Truncated or oversized >>> response headers received from daemon process 'django' >>> >>> when error 500 is returned. After reading the group and the doc, I added >>> >>> WSGIApplicationGroup %{GLOBAL} >>> to apache configuration, but the issue still occurs sometimes - not on >>> the same request. It sounds to mainly occur when the server is under load >>> (which makes it difficult to reproduce in house). >>> >>> If I read the group correctly, this is likely due to a crash of >>> mod_wsgi, and the best way to get this solved is to use gdb ; >>> unfortunately, as the issue occurs on deployed servers, it's pretty hard to >>> track this. Would setting CoreDumpDirectory and restarting apache after >>> setting ulimit -c unlimited help finding what's going wrong ? (running on >>> Centos 5 BTW). >>> >>> Any hint on what could we do to track the issue and figure out what's >>> causing the error ? >>> >>> Any help would be really appreciated, >>> Best regards, >>> Pierre-Nicolas >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. >> <mod_wsgi.txt.zip> >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] <javascript:> > . > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
