If want to get more serious with, have a read of: http://blog.dscpl.com.au/2015/06/implementing-request-monitoring-within.html <http://blog.dscpl.com.au/2015/06/implementing-request-monitoring-within.html>
There are APIs in mod_wsgi to extract metrics information. A key one in that is the thread utilisation and number of activated threads. They can help to tune the settings properly. The PyCon talk video was using that metrics API. Graham > On 5 Jun 2017, at 6:29 PM, Vishnu Prasad <[email protected]> wrote: > > Thanks Graham. I have put in place a plan to test with > 1ProcessX3Threads,2ProcessX3Threads,4ProcessX3Threads and likewise for > 6,10,20,40 threads. Again for me benchmark is the CPU usage with response > time. Anything above 75% CPU, will try to mark it as a comfort zone....in > addition planning to review how load average varies with the above. > > Will share what i end up.. > > thanks > > > > On Monday, June 5, 2017 at 6:26:49 AM UTC+5:30, Graham Dumpleton wrote: > >> On 5 Jun 2017, at 2:04 AM, Vishnu Prasad <[email protected] <>> wrote: >> >> Thanks Graham. Got my code ready @ >> https://github.com/vishnuprasadh/flaskdynamicimgprocessor >> <https://github.com/vishnuprasadh/flaskdynamicimgprocessor>. Will Perf test >> and update, was a very wise guidance from you. > > Depending on how the image library being used is implemented, using a high > number of threads as you have in: > > WSGIDaemonProcess imgprocessor user=vph home=/var/www/imgprocessor > threads=40 > > may be a bad idea. > > The problem is that the global interpreter lock (GIL) in Python means that > only one piece of Python code can run at a time. > > If the image library being used isn't able to release the GIL while doing the > image resize, and use C level operations only which don't need to lock the > Python object, then you can't have multiple image resize operations running > in parallel. Being a CPU bound activity also doesn't help your course. > > For CPU bound activities, especially where the GIL cannot be released and the > operation can take a while, you should favour use of multiple processes over > threads as then you can get proper concurrency by virtue of the CPU intensive > task running on separate processes. > > Use of a high number of threads should only be done when you have a I/O bound > application. Even then, running 40 threads rings alarm bells. Usually you > still want to run a lot less than that if can. > > Also, as far as testing performance, ensure you aren't testing by simply > overloading your web server. That is completely the wrong way of going about > it. > > I would recommend you only supply as much traffic as would cause a single web > application process to be within the 40-60% CPU range if using Python code > and so limited by the GIL. If you are reaching more than that, you should be > creating more processes. > > Note that even if you put threads at 40 and do overload your application, it > doesn't necessarily mean that all threads are being used. Most likely in a > CPU intensive application, most threads sit there unused and so wasting a bit > of memory. That is why need to monitor CPU usage of processes, but also the > thread utilisation factor. > > I did a talk about this at regional PyCon some time back. See: > > * https://www.youtube.com/watch?v=SGleKfigMsk > <https://www.youtube.com/watch?v=SGleKfigMsk> > > Lastly, neither of these is needed in the configuration: > > WSGIScriptReloading On > Options +ExecCGI > > Script reloading is on by default. ExecCGI is only needed if using > AddHandler, not when using WSGIScriptAlias. > > And: > > Order allow,deny > Allow from all > Require all granted > > is using both Apache 2.2 and 2.4 methods of granting access. I wouldn't show > both. If need to, use: > > <IfVersion < 2.4> > Order allow,deny > Allow from all > </IfVersion> > > <IfVersion >= 2.4> > Require all granted > </IfVersion> > > Graham > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
