Tried it and no luck, same issue, seems to happen more often on the production machine which is also behind a load balancer, got the process stuck there on the first try with the config change. Also, the process doesn't seem be in zombie status neither, looks completely like a normal process (and it probably is since background threads stay running) but isn't receiving requests. I can't really understand how can a process stay alive like this and stay running normally even after a few sigterm and sigkill signals!
What's odd is that the process id that is "stuck" is not really the one that was attempted to be killed, but I'm not familiar with mod_wsgi/apache internals so that's probably fine, below is both the two stuck processes from top and the logs. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ RUSER COMMAND 5948 wsgi 20 0 1203m 88m 11m S 0.3 4.4 0:03.40 wsgi httpd 12673 wsgi 20 0 1210m 146m 11m S 0.3 7.3 11:26.47 wsgi httpd ---> this one should not be here. [Sat Dec 31 21:38:43.097424 2016] [core:warn] [pid 12669:tid 140513528571968] AH00045: child process 13723 still did not exit, sending a SIGTERM [Sat Dec 31 21:38:45.099655 2016] [core:warn] [pid 12669:tid 140513528571968] AH00045: child process 13723 still did not exit, sending a SIGTERM [Sat Dec 31 21:38:47.101924 2016] [core:warn] [pid 12669:tid 140513528571968] AH00045: child process 13723 still did not exit, sending a SIGTERM [Sat Dec 31 21:38:49.104142 2016] [core:error] [pid 12669:tid 140513528571968] AH00046: child process 13723 still did not exit, sending a SIGKILL [Sat Dec 31 21:38:50.812271 2016] [suexec:notice] [pid 5944:tid 140156604848192] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sat Dec 31 21:38:50.825993 2016] [auth_digest:notice] [pid 5944:tid 140156604848192] AH01757: generating secret for digest authentication ... [Sat Dec 31 21:38:50.826665 2016] [lbmethod_heartbeat:notice] [pid 5944:tid 140156604848192] AH02282: No slotmem from mod_heartmonitor [Sat Dec 31 21:38:50.827032 2016] [:warn] [pid 5944:tid 140156604848192] mod_wsgi: Compiled for Python/2.7.9. [Sat Dec 31 21:38:50.827041 2016] [:warn] [pid 5944:tid 140156604848192] mod_wsgi: Runtime using Python/2.7.10. [Sat Dec 31 21:38:50.827503 2016] [core:warn] [pid 5944:tid 140156604848192] AH00098: pid file /var/run/httpd/httpd.pid overwritten -- Unclean shutdown of previous Apache run? [Sat Dec 31 21:38:50.828766 2016] [mpm_event:notice] [pid 5944:tid 140156604848192] AH00489: Apache/2.4.23 (Amazon) mod_wsgi/3.5 Python/2.7.10 configured -- resuming normal operations [Sat Dec 31 21:38:50.828782 2016] [core:notice] [pid 5944:tid 140156604848192] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' El sábado, 31 de diciembre de 2016, 1:27:46 (UTC-3), Graham Dumpleton escribió: > > > On 31 Dec 2016, at 3:07 PM, Cristiano Coelho <[email protected] > <javascript:>> wrote: > > Hello, > > So the configuration might not be ideal, but these small config tweaks > shoun't be really the source of the issue right? It's a bit odd since I > haven't had issues with similar deploys. This project I guess uses more > libraries C extensions (lxml and postgres) and the background threads/pools > perform a lot of IO (email sending, message queue polling, and some > others), although they are all under daemon threads and should finish under > the 4 seconds grace time. > > > If you are using lxml then you definitely need to use: > > WSGIApplicationGroup %{GLOBAL} > > as from memory it is one of the libraries which is known to have issues > when used in Python sub interpreters. The problem will be if a callback > function is registered which lxml calls when parsing XML. Because it > doesn’t deal with thread locking properly when using a sub interpreter, it > can deadlock its own thread. Other threads can still run, but if other > request threads do the same, you can eventually exhaust all the request > threads and the process hangs. Background threads you create separately > could still run though. Although even if this occurs, it shouldn’t stop an > Apache restart from killing the process. > > Would setting WSGIApplication Group %{GLOBAL} still allow me to use more > than 1 process on the daemon configuration? Although I don't think it will > do any change at all since the web servers only listen on port 80 and are > on the same domain so all requests should always be falling into the same > application group if I interpreted the docs correctly. > > > Application group is the Python interpreter context within each respective > process. The value %{GLOBAL} just means the main or first interpreter > context of the process. This is the same as if you had run command line > Python and behaves the same. Any additional interpreter contexts created in > a process are what are referred to as sub interpreter contexts. By default > mod_wsgi uses a separate sub interpreter context in each process for each > WSGI application delegated to run in the same set of processes. > > So there is no restriction on setting ‘processes’ option of > WSGIDaemonProcess to be more than one at the same time as setting > WSGIApplicationGroup to %{GLOBAL}. > > This issue is so random and since only happens on cloud deploys it gets > really difficult to test if a change helped or not and it can take days to > notice it. I guess I will keep playing around with settings and try to > gather more info of the stuck processes when it happens. > > > Which sounds even more like issue with sub interpreters. If the bit of > code which triggers the deadlock is infrequent, then the loss of request > threads could be slow. This is where newer mod_wsgi versions at least have > various timeout options for causing daemon process restarts when requests > timeout or block. > > At the least, add: > > WSGIApplicationGroup %{GLOBAL} > > and see how it goes. > > Graham > > Thank you so much for all the help! > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] <javascript:> > . > Visit this group at https://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
