Graham, Thanks a ton for the response. I have updated more inputs inline for your review and suggestions.
thanks On Friday, April 7, 2017 at 8:57:22 AM UTC+5:30, Graham Dumpleton wrote: > > > On 7 Apr 2017, at 9:40 AM, Vishnu Prasad <[email protected] > <javascript:>> wrote: > > All, > I am new to mod_wsgi and the entire configurations around related to this. > Detailing out the requirement, please do let me know your response. > > 1. I have written a simple python script to handle images which listens on > port 8041. There is some custom processing happening here and its not just > static rendering. The output of this is a content of type image and bytes > is the response. > 2. I am starting this python script file(application) using command > mod_wsgi-express start-server abc.py --threads 20 --process 2 > > > Are you doing image processing? If you are and it is CPU intensive, then > use of threads, rather than processes, isn't then best idea. > [Vishnu:] Yes, am using Pillow-SIMD to dynamically generate or resize images for optimize image rendering. Once returned from here, caching in varnish. We have over 100,000 items to process and at any given point easily and i will need to generate about 20-30 images/sec. This webserver cluster is about 4 and sitting behind LB. I have done image domain randomization to leverage capability of browsers to download more parallel images. Created about 3 subdomains for this as well, hence processes of 2. Agree - Once performance tested we may tune this further. Further this webserver is dedicated to passthrough request to next level appserver outside of which this is the only CPU intense process working on these. > > 3. I have varnish script that whenever gets any requests for images on > 8080 routes to this application on port 8041. > 4. This application is also running another default instance of apache2 > which is serving my all other application requests not related to images. > > > And this is a Python application also hosted on mod_wsgi, but configured > manually in normal Apache instance rather than using mod_wsgi-express? > [Vishnu:] Yes, this is on mod_wsgi but not configured through normal > apache instance. As you said in one of your blog, getting it working was > first - so started in embedded mode and now reviewing if I should move out > or keep it that way for one set of URL's. This means all images goes to > mod_wsgi-express and others go to the normal apache default instance. > Earlier I thought of having a virtualhost config to route this for all > /image patterns i.e. Daemon config as you said in one of the blog, later > realized about the modularity of this setup. I wanted to use this as > stand-alone modular imageserver. I am reading through your link below on > the issues with embedded mode of running. I will review both these options > and run some performance tests. This is helpful. > > The above setup is working. > > Problem/Help needed: > 1. I want to configure and run the above application as distinct > WSGIDaemonProcess. How do I achieve this ? > > > If for (4) you are running a Python application hosted on mod_wsgi, you > should already be using daemon mode as embedded node is generally not > recommended as explained in: > > > http://blog.dscpl.com.au/2012/10/why-are-you-using-embedded-mode-of.html > <http://www.google.com/url?q=http%3A%2F%2Fblog.dscpl.com.au%2F2012%2F10%2Fwhy-are-you-using-embedded-mode-of.html&sa=D&sntz=1&usg=AFQjCNGiURKOSUfshe-g4r-68VMGRnLOYQ> > [Vishnu:] I am running this app in embedded mode as i was planning to use this more to manage/handle only image requests. > > 2. If the above ask is accomplished by a configuration in default apache > httpd.conf (not mod_wsgi httpd.conf) then i also want to know how will > apache identify my application vs other applications of mod_wsgi which may > be running ? Is it based on server url or ? > > > How to vertically partition a URL namespace so as to directory subsets of > URLs to a different daemon process group within the same Apache instance, > is explained in: > > > http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html > <http://www.google.com/url?q=http%3A%2F%2Fblog.dscpl.com.au%2F2014%2F02%2Fvertically-partitioning-python-web.html&sa=D&sntz=1&usg=AFQjCNFSqPozGdQVlEtWMQUYSzWo1LEzOQ> > > 3. For the above application, need to have better control over what is > loaded in httpd.conf. So, want to load my own httpd.conf including path of > log mounts etc. How do I achieve this ? > > [Vishnu:] My question was how do i specifically setup a httpd.conf for > this embedded mode i.e. outside the core apache2 installation. Based on > what i read from you its better to not run in embedded mode. However, if > we need to then the configuration around logs for this specifically will > need to happen in mod_wsgi hence the ask. > Again, am not generating apache config instead am asking how do i tell mod_wsgi-express, instead of you reading from /tmp/....httpd.conf, read this from /usr/local/httpd.conf ? Is there some server-base path setting which i could use and set this or ? I completely understand your point and dont want to mess up over-engineering the solution. > > I don't understand what you mean by this. > > You should not be using mod_wsgi-express to generate Apache configuration > to then use in an existing Apache installation. The mod_wsgi-express > instance is self contained with a generated configuration. If > customisations for that are required, most common things can be overridden > using the command line options. For some special cases you can use > --include-file to add additional configuration at the end of the generated > configuration if necessary. > 4. Finally, at somepoint if I want to manage the httpd.conf common file > across WSGI and my default apache, how do I set up the application specific > configurations - through virtualhost section in httpd-vhost.conf or ? > > > If for the other (4) above the answer is that you are also using a > manually configure Apache/mod_wsgi, as well as a separate mod_wsgi-express > for just one subset of URLs, then as per blog on vertically partition, you > probably should just do it all manually in one Apache/mod_wsgi instance if > want to have config all in one spot. Personally I would probably use > multiple mod_wsgi-express instance sitting directly behind varnish or > whatever front end you are using. The generated configuration from > mod_wsgi-express is usually going to be better than what many would set up > manually as it sets a whole lot of good stuff related to timeouts which > makes things more robust. When Apache/mod_wsgi is manually configured there > are no default timeouts set. > [Vishnu:] Understand that you suggest against embdedded mode and against managing 2 apache conf files. Even if we need, then you suggest me to take just mod_wsgi-express. I will review this as said earlier as well. Also, i went through the httpd.conf of mod_wsgi process for WSGIDaemon node and its quite extensive. I will keep you posted on the outcome. thanks a ton for a detailed answer, the links are helpful and am reading through them for guidance. > > Anyway, suggest you read that blog on vertically partition a web > application at least and clarify things by answering questions above. > > Graham > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
