On Sat, Aug 23, 2008 at 1:33 PM, Graham Dumpleton <[EMAIL PROTECTED]> wrote: > > 2008/8/22 Nimrod A. Abing <[EMAIL PROTECTED]>: >>>>> Important changes are file descriptor leaks on a graceful restart in >>>>> both versions and for version 2.2 possible truncation of data when >>>>> using wsgi.file_wrapper. An upgrade would be recommended especially >>>>> due to the latter if using wsgi.file_wrapper in a way or on system >>>>> affected by the problem. >>>> >>>> Thanks for the fix for wsgi.file_wrapper. We had been experiencing >>>> problems lately with users uploading large files. All tracebacks seem >>>> to point to something in wsgi.file_wrapper. Unfortunately it is one of >>>> those "heisenbugs". Some random thing/file upload triggers it, it >>>> disappears when you try to debug it and it's impossible to recreate. >>> >>> You mean downloading don't you? At least I take uploading to mean >>> putting a file on server, not the other way around, so wouldn't expect >>> wsgi.file_wrapper to be involved. I would also have expected you to be >>> using Apache 2.2 on UNIX where for static file downloads wouldn't be >>> an issue. >> >> Uploading usually. But I have reason to suspect that it's happening >> because users abort the upload. > > Which can cause 'IOError: client connection closed' exception to be > raised when attempting to read input. > >> But we also get weird tracebacks from >> time to time, these involved URL's where nothing is POSTed. We are >> currently using Apache 2.0. We use lighttpd for static content. > > Am curious to see what these might be and whether they might be > originating in mod_wsgi.
Looking back at the majority of tracebacks I have received so far, all of them are 'IOError: client connection closed' and all of them are for the URL used for posting uploads. There is the odd exception for URL's but I did some further digging and correlated them to the numerous "probes" that we are getting. So this might very well have nothing to do with mod_wsgi, though I will still keep an eye out for the issues that 2.2 has fixed. We are using it in production now and so far everything has been going along swimmingly. >>> Maybe you want to better explain the problem you are seeing in case it >>> is something completely different. One concern would be that if it is >>> a large file upload that takes more than 5 seconds, that if you are >>> using daemon mode and recycling process based on number of requests, >>> that upload is being interrupted when process is recycled, as by >>> default will only allow at most 5 seconds for process to restart >>> before killing it. In other words, it is not a graceful restart where >>> process persists until all active requests complete. >> >> We are using daemon mode. It's interesting that you brought that up. >> It had never occurred to me that the daemon could timeout in the >> middle of a POST request, in any case I would not expect it to timeout >> in the middle of *any* request. Like I said above, it happens more >> often on uploads. I suspect that these users are using dial-up, a lot >> of the IPs are from Cox address block. What are the defaults for the >> *-timeout options if you do not supply them? > > The timeouts only come into play when you use 'maximum-requests' > option to WSGIDaemonProcess. It is defined by 'shutdown-timeout' and > if not defined defaults to 5 seconds. What occurs is that when maximum > requests is reached, the process starts an orderly shutdown sequence > of not accepting new requests, allowing running requests to complete > and then triggering atexit registered callbacks and destroying Python > interpreter instances. If however the active requests do not complete > within that shutdown timeout period, then the process is killed off > without the cleanup occurring. Our config does not have maximum-requests specified. You do bring up a very good way of dealing with scaling issues that we are anticipating. > Thus, a long running request can prevent restart of a daemon process > occurring immediately and can also prevent proper cleanup occurring. > > If using 'maximum-requests' it is also a good idea to be running > multiple daemon processes, because if you are only running one, a long > running request can effectively stall the application for that period, > with users seeing a delay. By using multiple daemon processes, when > one process is restarting because maximum requests reached, other > daemon processes can be handling the requests. You could still get all > daemon processes restarting at same time, but chances reduce the more > you have. > > Now, to answer the question as to why the request isn't allowed to > complete, it is because I believe that is one of the issues which > causes some fastcgi implementations to leave processes around and not > kill them off properly. Thus mod_wsgi takes the approach of always > ensuring that nothing can stall things completely or leave phantom > processes around. Makes sense to me. For one, I don't want my boss to be calling me in the middle of the night because the server has stalled :) >>> Thus, long running file uploads perhaps better handled in embedded >>> mode, or delegate the URLs for file upload to special daemon process >>> that doesn't recycle process so as to avoid problem with it being >>> interrupted. >> >> That would require us to have a subdomain just for this purpose, I'm >> predicting this will cause the Django authentication system to fail >> somehow. But I will look into it and see if it's feasible. > > No. You start on right track in later messages, but will answer here. > > <VirtualHost *> > > WSGIDaemonProcess default processes=5 threads=10 maximum-requests=5000 > WSGIDaemonProcess uploads processes=1 threads=10 > > WSGIProcessGroup default > > WSGIScriptAlias / /some/path/app.wsgi > > <Locaton /my/url/for/uploads> > WSGIProcessGroup uploads > </Location> > > ... > > </VirtualHost> > > In other words, using <Location> directive with WSGIProcessGroup > within it, you can distribute URLs across multiple daemon process > groups with different attributes such as number of processes, maximum > requests etc. The WSGIProcessGroup inside of <Location> directive will > override that defined at scope of <VirtualHost> but only for matched > URL prefix. Excellent! We will probably be using a separate subdomain for handling file uploads. This will allow us to scale horizontally as well as vertically when the need arises. Good to know there is an option if you are unable to use subdomains. -- Best Regards, Nimrod A. Abing W http://arsenic.ph/ W http://preownedcar.com/ W http://preownedbike.com/ W http://abing.gotdns.com/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en -~----------~----~----~----~------~----~------~--~---
