Personally I would be concerned about the architecture you are using if you have long running tasks like you describe. It is not usually a good idea to use 'multiprocessing.Process' to create sub processes directly from a web server process to perform work. A better architecture would be to off load the work into a queue using something like Celery and have the separate job processing system pull the jobs from the queue and process them. You would also be better off to model the interaction from the front end as queueing the job and immediately responding with an acknowledgement to say is queued. The front end can then start polling periodically to see if the job has finished, and when it has it would get the response back. The front end can then display the data or save it locally as needed.
This model avoids the problem of requests being parked doing nothing for a long time, which with your server configuration is going to be hugely expensive on memory and not scale very well because of limitations of using WSGI process/threading model. You might even consider not using a WSGI application at all. Instead, use an async web application paired with Celery for execution of the jobs. Using an async web application means you can handle as many parked requests as you want and they can quite happily sit there waiting for Celery to finish the job and don't need to use polling. Only thing am not sure about in that is what async clients there are for Celery. Graham > On 10 Aug 2020, at 9:09 pm, Paul Royik <[email protected]> wrote: > > My django app makes heavy calculations which can be infinite. > That's why, when user enters the site, i.e. makes a request, heavy > calculations are wrapped into multiprocessing.Process which runs at most 7 > seconds. > I can't use threads, because third-party packages are not thread-safe. > > So, I have around 30 concurrent requests per second. If each request can take > up to 7 seconds, then it is 30*7=210 concurrent requests in the worst case. > Each of these concurrent requests opens multiprocessing.Process, which gives > (I guess) 210*2=420 (close to 500) concurrent requests in the worst case. > That' how I got 500 requests. Possibly, my calculations are incorrect. > > Average page load time (average response times) is 10 seconds. > > I use MPM worker. > > I set WSGIProcessGroup > > StartServers 100 > ServerLimit 500 > > ThreadsPerChild 1 > ThreadLimit 1 > > MaxRequestWorkers 500 > MaxConnectionsPerChild 10000 > > WSGIApplicationGroup %{GLOBAL} > WSGIDaemonProcess django_app processes=75 threads=1 python-path='...' > maximum-requests=10000 request-timeout=20 > WSGIProcessGroup django_app > WSGIRestrictEmbedded On > WSGILazyInitialization On > > > On Monday, August 10, 2020 at 1:12:30 PM UTC+3, Graham Dumpleton wrote: > What sort of application are you running? > > What is your average response times? > > Do you have long running requests, if yes, how long? > > What Apache MPM are you actually using? > > My initial impression is that is a quite poor configuration which is only > going to chew up huge amounts of memory for no good reason, but I don't know > your application requirements. > > Also, are you even setting WSGIProcessGroup? If it isn't set it makes the > whole daemon process configuration moot as it isn't even being used. > >> On 10 Aug 2020, at 7:24 pm, Paul Royik <distan...@ <>gmail.com >> <http://gmail.com/>> wrote: >> >> StartServers 50 >> ServerLimit 200 >> >> ThreadsPerChild 1 >> ThreadLimit 1 >> >> MaxRequestWorkers 200 >> MaxConnectionsPerChild 10000 >> >> WSGIApplicationGroup %{GLOBAL} >> WSGIDaemonProcess process processes=75 threads=1 >> >> >> Is it enough? Or can it handle only 75 concurrent requests? I don't know how >> to synchronize apache and mod_wsgi settings. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to mod...@ <>googlegroups.com <http://googlegroups.com/>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/F668B434-7E86-4719-90AE-2B472EDC94BE%40gmail.com.
