If it is a minor quick running script that does something simple it should be okay. It is just having long running processes would be more concerned about.
> On 11 Aug 2020, at 3:18 pm, Paul Royik <[email protected]> wrote: > > You are absolutely correct. Need to change the architecture. > One more question. I also use subprocess.check_output from django. Is it also > bad idea? I'm trying to run a script (non-python) and get it output. > > On Tuesday, August 11, 2020 at 1:55:51 AM UTC+3, Graham Dumpleton wrote: > Personally I would be concerned about the architecture you are using if you > have long running tasks like you describe. It is not usually a good idea to > use 'multiprocessing.Process' to create sub processes directly from a web > server process to perform work. A better architecture would be to off load > the work into a queue using something like Celery and have the separate job > processing system pull the jobs from the queue and process them. You would > also be better off to model the interaction from the front end as queueing > the job and immediately responding with an acknowledgement to say is queued. > The front end can then start polling periodically to see if the job has > finished, and when it has it would get the response back. The front end can > then display the data or save it locally as needed. > > This model avoids the problem of requests being parked doing nothing for a > long time, which with your server configuration is going to be hugely > expensive on memory and not scale very well because of limitations of using > WSGI process/threading model. You might even consider not using a WSGI > application at all. Instead, use an async web application paired with Celery > for execution of the jobs. Using an async web application means you can > handle as many parked requests as you want and they can quite happily sit > there waiting for Celery to finish the job and don't need to use polling. > Only thing am not sure about in that is what async clients there are for > Celery. > > Graham > >> On 10 Aug 2020, at 9:09 pm, Paul Royik <distan...@ <>gmail.com >> <http://gmail.com/>> wrote: >> >> My django app makes heavy calculations which can be infinite. >> That's why, when user enters the site, i.e. makes a request, heavy >> calculations are wrapped into multiprocessing.Process which runs at most 7 >> seconds. >> I can't use threads, because third-party packages are not thread-safe. >> >> So, I have around 30 concurrent requests per second. If each request can >> take up to 7 seconds, then it is 30*7=210 concurrent requests in the worst >> case. >> Each of these concurrent requests opens multiprocessing.Process, which >> gives (I guess) 210*2=420 (close to 500) concurrent requests in the worst >> case. >> That' how I got 500 requests. Possibly, my calculations are incorrect. >> >> Average page load time (average response times) is 10 seconds. >> >> I use MPM worker. >> >> I set WSGIProcessGroup >> >> StartServers 100 >> ServerLimit 500 >> >> ThreadsPerChild 1 >> ThreadLimit 1 >> >> MaxRequestWorkers 500 >> MaxConnectionsPerChild 10000 >> >> WSGIApplicationGroup %{GLOBAL} >> WSGIDaemonProcess django_app processes=75 threads=1 python-path='...' >> maximum-requests=10000 request-timeout=20 >> WSGIProcessGroup django_app >> WSGIRestrictEmbedded On >> WSGILazyInitialization On >> >> >> On Monday, August 10, 2020 at 1:12:30 PM UTC+3, Graham Dumpleton wrote: >> What sort of application are you running? >> >> What is your average response times? >> >> Do you have long running requests, if yes, how long? >> >> What Apache MPM are you actually using? >> >> My initial impression is that is a quite poor configuration which is only >> going to chew up huge amounts of memory for no good reason, but I don't know >> your application requirements. >> >> Also, are you even setting WSGIProcessGroup? If it isn't set it makes the >> whole daemon process configuration moot as it isn't even being used. >> >>> On 10 Aug 2020, at 7:24 pm, Paul Royik <distan...@ <>gmail.com >>> <http://gmail.com/>> wrote: >>> >>> StartServers 50 >>> ServerLimit 200 >>> >>> ThreadsPerChild 1 >>> ThreadLimit 1 >>> >>> MaxRequestWorkers 200 >>> MaxConnectionsPerChild 10000 >>> >>> WSGIApplicationGroup %{GLOBAL} >>> WSGIDaemonProcess process processes=75 threads=1 >>> >>> >>> Is it enough? Or can it handle only 75 concurrent requests? I don't know >>> how to synchronize apache and mod_wsgi settings. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to mod...@ <>googlegroups.com <http://googlegroups.com/>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to mod...@ <>googlegroups.com <http://googlegroups.com/>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CAC5F7F3-52D3-4A8A-91FA-D9E08E9C29BB%40gmail.com.
