Hi again. Finally I could resolve my problem early importing urls.py (which imports indirectly a lot of other modules) before uwsgi forks. The problems seems to be that sometimes two or more concurrent threads get deadlocked during initial imports (perhaps due to mutual / circular dependencies). I don't know if it could have an easy fix.
The solution has some benefits: the uwsgi workers share much more code with the master (so that overall memory consumption is less), the first request is processed much faster, it requires less work with initial imports after a worker reload. I have update the GH issue. Should I close it? Best regards. Ernesto Revilla Área Técnica TangramBPM.es Tlf: 630 244 136 2017-10-11 10:50 GMT+02:00 Ernesto Revilla <erevi...@tangrambpm.es>: > Hi. > > Thanks for helping with this issue. Here some more notes (already updated > in the GitHub issue) > > * On OS X, I get high CPU load with I use --thunder-lock options > * On Linux, I recompiled uwsgi with posix_sem semaphores locking > mechanism, but I still get a futex wait. > * The error seems to be related to concurrent imports. It's much like > there are any circular imports, but they only raise the issue if they are > run from two threads simultaneously. > * I run additional imports (code preloading) in django.wsgi script. This > makes forks much more efficient as more code is shared. But database and > memcached connections must be closed before uwsgi forks the master. > > Regards. > > > > > Ernesto Revilla > Área Técnica > TangramBPM.es > Tlf: 630 244 136 > > > > 2017-09-29 14:29 GMT+02:00 Ernesto Revilla <erevi...@tangrambpm.es>: > >> As suggested on the GH issue: >> https://github.com/unbit/uwsgi/issues/1599#issuecomment-332824601 >> >> I disabled "thunder-lock" but it doesn't correct the problem. >> >> Any suggestions? >> >> What I have in mind is test with more recent Python versions (>= 2.7.12). >> >> Best regards. >> >> >> >> Ernesto Revilla >> Área Técnica >> TangramBPM.es >> Tlf: 630 244 136 >> >> >> >> 2017-09-28 13:23 GMT+02:00 Ernesto Revilla <erevi...@tangrambpm.es>: >> >>> I use max-requests = 450. After the first request, the process has >>> aprox. 160MB. >>> >>> After more tests, it seems to have to do with the first requests to an >>> reloaded idle instance. It may be a concurrency problem, e.g. concurrent >>> imports. I think I have captured this situation with strace, but I'm still >>> looking at it. >>> >>> Regards. >>> >>> Ernesto Revilla >>> Área Técnica >>> TangramBPM.es >>> Tlf: 630 244 136 >>> >>> >>> >>> 2017-09-28 9:12 GMT+02:00 Avraham Serour <tovm...@gmail.com>: >>> >>>> if you are having probems with memory leaks I suggest tweaking the 1k >>>> request worker reload setting (I forgot what it is called right now) >>>> This won't solve the issue but can lower the impact of the problem >>>> until it is solved >>>> >>>> how much does it use after initialization? after the first request? >>>> >>>> Do you have 1k requests total every 30-60 minutes or is this what each >>>> worker gets every 30-60 minutes? >>>> >>>> >>>> On Thu, Sep 28, 2017 at 9:46 AM, Ernesto Revilla < >>>> erevi...@tangrambpm.es> wrote: >>>> >>>>> Hi. >>>>> >>>>> No, at least not at the production site, because memory consumption >>>>> (we still have memory leaks) and application initialization is too high >>>>> and >>>>> each process gets > 450 MB after about 1000 requests (aprox. 30-60 min). >>>>> The VM has 4GB RAM. Now I have 4 workers. >>>>> >>>>> Currently, I'm not able to reproduce the error reliably. >>>>> >>>>> Regards. >>>>> >>>>> Ernesto Revilla >>>>> Área Técnica >>>>> TangramBPM.es >>>>> Tlf: 630 244 136 >>>>> >>>>> >>>>> >>>>> 2017-09-28 7:41 GMT+02:00 Avraham Serour <tovm...@gmail.com>: >>>>> >>>>>> did you try running it without threads? >>>>>> >>>>>> On Wed, Sep 27, 2017 at 5:26 PM, Ernesto Revilla < >>>>>> erevi...@tangrambpm.es> wrote: >>>>>> >>>>>>> Dear all. >>>>>>> >>>>>>> Could anyone please lend me a hand? >>>>>>> >>>>>>> From time to time, the respawned (--reload-on-rss=450) process is >>>>>>> blocked and does not accept any connections. I get a "Connection timed >>>>>>> out" >>>>>>> error in Nginx. >>>>>>> >>>>>>> https://github.com/unbit/uwsgi/issues/1599 >>>>>>> >>>>>>> I did already strace it and find the process and all its threads >>>>>>> waiting on futex. >>>>>>> >>>>>>> I haven't read the "reload code", but it's like if the forked >>>>>>> process inherits some other futex (we can see it with strace). >>>>>>> >>>>>>> I've been able to reproduce the behaviour using Apache Bench (ab) on >>>>>>> a test server. I was not able to reproduce it on Ubuntu 16.04. >>>>>>> >>>>>>> Thanks in advance. >>>>>>> Best regards. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Ernesto Revilla >>>>>>> Área Técnica >>>>>>> TangramBPM.es >>>>>>> Tlf: 630 244 136 >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> uWSGI mailing list >>>>>>> uWSGI@lists.unbit.it >>>>>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> uWSGI mailing list >>>>>> uWSGI@lists.unbit.it >>>>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> uWSGI mailing list >>>>> uWSGI@lists.unbit.it >>>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> uWSGI mailing list >>>> uWSGI@lists.unbit.it >>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >>>> >>>> >>> >> >
_______________________________________________ uWSGI mailing list uWSGI@lists.unbit.it http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
